Exact Design Space Exploration Based on Consistent Approximations

Round 1

Reviewer 1 Report

This paper provides a DSE approach to approximate evaluations of experimental evaluations with a certain degree of error-rate. The experimental results show benefits of using the proposed approach, however, due to some missing parts, I'd recommend a revision to the current version.

Missing Citations:

- Iterative and Multi-objective DSE:
1) Palermo, Gianluca, Cristina Silvano, and Vittorio Zaccaria. "ReSPIR: a response surface-based Pareto iterative refinement for application-specific design space exploration." IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 28.12 (2009): 1816-1829.

2) Silvano, Cristina, et al. "Multicube: Multi-objective design space exploration of multi-core architectures." VLSI 2010 Annual Symposium. Springer, Dordrecht, 2011.

- System-level DSE:
3) Benini, Luca, et al. "Mparm: Exploring the multi-processor soc design space with systemc." Journal of VLSI signal processing systems for signal, image and video technology 41.2 (2005): 169-182.

4) Xie, Yuan, et al. "Design space exploration for 3D architectures." ACM Journal on Emerging Technologies in Computing Systems (JETC) 2.2 (2006): 65-103.

- Compiler-level DSE:
5) Ashouri, Amir Hossein, et al. "A framework for compiler level statistical analysis over customized vliw architecture." 2013 IFIP/IEEE 21st International Conference on Very Large Scale Integration (VLSI-SoC). IEEE, 2013.

Results:

One missing part in this paper is the lack of comparative results wrt to the existing works. Additionally, I'd like to thank the authors to provide their source-code online.

Section 4.3. Line 322: You need to provide details on why the assumption does not hold for real use-cases.

Figure 10. Explain the whiskers in the box plots, are they representing 25% and 75%? Also, the line in the box needs to be clarified whether or not ot was a median or mean. I still can't understand why there is such a gigantic difference between the low and the medium ERRs. You need to explain that in more depth to the users and explore possible remedies.

Figure 11. is showing a relative loss in the accuracy, you also need to report the absolute error-rate for your approach to the user. Due to the normalization policy, we need to know what the actual deltas were in evaluating your approach. How did you produce the lines here? Were they obtained by arithmetic mean or geometric mean? If you are averaging speedups, you would have needed to mean using geometric mean and if that was the case, please revise the results and report them in a separate table.

English Text Revisions:

The English text can benefit from a major revision. Here are some instances but you get the idea:

P1, Line 8: Cheap -> an efficient
P1, Line 8: in order to -> to
P1, Line 24: good -> the good
P1, Line 33: desired -> the desired
P3, line 79: has -> have
...
P4, line 135: "The earliest" -> "To the best of our knowledge, the earliest ..."
...
P18, line
P18, Line 615: "The results" is vague to the user -> "Figure 11. depicts our experimental results" ... .

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

The paper addresses an important concern to design efficient systems. However, it can be improved by addressing the following issues.

 

In the related works, it would be good to distinguish the proposed work with respect to RAPIDITAS [13] that also used approximations. Similarly, approximations are used in [18], [19] and others. It would good to make a summary of distinction to show the novelty of the proposed approach. Some comments at the end of the related works are provided to show the difference in the end results, but they should be brought forward and distinction of the working of approaches should be mentioned leading to different types of results.

 

There are some approximation-based approaches that employ individual application profiling and get the design point for multiple applications by analysing these individual profiles, e.g. Resource and Throughput Aware Execution Trace Analysis for Efficient Run-time Mapping on MPSoCs. These should be included and discussed.

 

Good set of background are provided.

 

It is unclear how the proposed approach ensures optimal solutions. A more thorough discussion is needed and possibly can be proved mathematically.

 

The evaluation is weak. The results obtained with the proposed approach needs to be compared with existing relevant approaches considering approximations or it should be stated why a comparison is not possible. In particular, the accuracy, speed (DSE time) and solution quality comparisons are needed. 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

This paper proposes an approximation-based technique for accelerating the design space exploration task. The approach manages to achieve a significant reduction in the design space without sacrificing the correctness and completeness of the proposed solutions.

In general, the paper seems complete in all aspects. It is a well-written paper with a good background information. The authors discuss thoroughly not only the advantages, but also the limitations of their approach.

The whole approach is useful and practical but the authors should evaluate more complex systems and applications. Defining and evaluating the term “safe approximations” in more complex scenarios will be challenge.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 4 Report

In this work, the authors propose a novel model to optimise Design Space Exploration (DSE) applies to embedded systems.

 

The novelty of this work is the use of evaluation-related approximations in DSE to obtain an optimal solution based on the Pareto front.

The authors describe (in the related work section) various alternatives that can be found in the bibliography, pointing out the key elements in their approaches.

The simulations based on the execution of SystemC can be quite heavy, but they can be executed in reasonable computation times.

The model has some usage restrictions such as a safe approximation of complex objective functions, or that have concentrated the design on an approach for evaluating latency.

I recommend checking small grammatical errors.

Overall, this paper is a meaningful work.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

Most of my comments are addressed, but I was expecting a comparative result between proposed and relevant existing approaches in terms of speed, solution quality and accuracy. The justification why only the proposed approach was evaluated should be added somewhere in the experimental setup. 

Author Response

Thank you for your comment. We fully concur that the justification would be beneficial in the experimental setup. Therefore, we added a paragraph at the beginning of Section 6 (page 16, indicated in red) that addresses this issue.

Thanks again for your valuable suggestions.