dotfiles/notes/efficiency-of-algorithms.org

#+TITLE: Efficiency of Algorithms
#+AUTHOR: Christopher James Hayward

#+HUGO_BASE_DIR: ~/.local/source/website
#+HUGO_SECTION: notes

+ Algorithms must be analyzed to determine resource usage
+ Efficiency is analogous to engineering productivity for a repeating process
+ Measured in complexity of time and space (resources)

* Background

Importance of efficiency with respect to time was first emphasized by *Ada Lovelace* in 1843 regarding the babbage machine:

#+begin_quote
In almost every computation a great variety of arrangements for the succession of the process is possible, and variouis considerations must influence the selections amongst them for the purpose of a calculating engine. One essential object is to chose that arrangement which shall tend to reduce to a minimum the time necessary for completing the calculation.
#+end_quote

* Analysis

The most commonly used notation to describe resource consumption (complexity) is using *Donald Knuth's* =Big O= notation.

| Notation     | Name         | Example                                                        |
|--------------+--------------+----------------------------------------------------------------|
| O(1)         | Constant     | Finding the median in a sorted list of measurements            |
| O(log n)     | Logarithmic  | Finding an item in an sorted array with a binary search        |
| O(n)         | Linear       | Finding an item in an unsorted array                           |
| O(n log n)   | Linearithmic | Performing a fast sorting algorithm                            |
| O(n²)        | Quadratic    | Multiplying two n digit numbers by a simple algorithm          |
| O(n²), c > 1 | Exponential  | Finding the optimal solution to the traveling salesman problem |

* Measurement

Expression as a function of the size of the input ~n~, with the main measurements being:

+ Time :: How long does the algorithm take to complete?
+ Space :: How much working memory is required?

Other measurements for devices whose power needs to be factored in, such as for battery operated devices or super computers:

+ Direct power :: How much power is needed to compute the algorithm?
+ Indirect power :: How much power is needed for cooling, lighting, etc?

Less commonly used:

+ Transmission size :: How much data must be transferred for operation?
+ External space :: Space needed physically (such as on disk)
+ Response time :: Relevant in real time applications
+ Total cost :: Cost of hardware dedication

** Time

+ Uses time complexity analysis
+ Measured in CPU time usage
+ Detailed estimates needed to compare performance
+ Difficult to measure in parallel processing

** Space

Concerned with the usage of memory resources, typically the memory or storage needed to hold:

+ Code for the algorithm
+ Working space
+ Input data
+ Output data
+ Processing data