compress-offload.rst 15 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328
  1. =========================
  2. ALSA Compress-Offload API
  3. =========================
  4. Pierre-Louis.Bossart <pierre-louis.bossart@linux.intel.com>
  5. Vinod Koul <vinod.koul@linux.intel.com>
  6. Overview
  7. ========
  8. Since its early days, the ALSA API was defined with PCM support or
  9. constant bitrates payloads such as IEC61937 in mind. Arguments and
  10. returned values in frames are the norm, making it a challenge to
  11. extend the existing API to compressed data streams.
  12. In recent years, audio digital signal processors (DSP) were integrated
  13. in system-on-chip designs, and DSPs are also integrated in audio
  14. codecs. Processing compressed data on such DSPs results in a dramatic
  15. reduction of power consumption compared to host-based
  16. processing. Support for such hardware has not been very good in Linux,
  17. mostly because of a lack of a generic API available in the mainline
  18. kernel.
  19. Rather than requiring a compatibility break with an API change of the
  20. ALSA PCM interface, a new 'Compressed Data' API is introduced to
  21. provide a control and data-streaming interface for audio DSPs.
  22. The design of this API was inspired by the 2-year experience with the
  23. Intel Moorestown SOC, with many corrections required to upstream the
  24. API in the mainline kernel instead of the staging tree and make it
  25. usable by others.
  26. Requirements
  27. ============
  28. The main requirements are:
  29. - separation between byte counts and time. Compressed formats may have
  30. a header per file, per frame, or no header at all. The payload size
  31. may vary from frame-to-frame. As a result, it is not possible to
  32. estimate reliably the duration of audio buffers when handling
  33. compressed data. Dedicated mechanisms are required to allow for
  34. reliable audio-video synchronization, which requires precise
  35. reporting of the number of samples rendered at any given time.
  36. - Handling of multiple formats. PCM data only requires a specification
  37. of the sampling rate, number of channels and bits per sample. In
  38. contrast, compressed data comes in a variety of formats. Audio DSPs
  39. may also provide support for a limited number of audio encoders and
  40. decoders embedded in firmware, or may support more choices through
  41. dynamic download of libraries.
  42. - Focus on main formats. This API provides support for the most
  43. popular formats used for audio and video capture and playback. It is
  44. likely that as audio compression technology advances, new formats
  45. will be added.
  46. - Handling of multiple configurations. Even for a given format like
  47. AAC, some implementations may support AAC multichannel but HE-AAC
  48. stereo. Likewise WMA10 level M3 may require too much memory and cpu
  49. cycles. The new API needs to provide a generic way of listing these
  50. formats.
  51. - Rendering/Grabbing only. This API does not provide any means of
  52. hardware acceleration, where PCM samples are provided back to
  53. user-space for additional processing. This API focuses instead on
  54. streaming compressed data to a DSP, with the assumption that the
  55. decoded samples are routed to a physical output or logical back-end.
  56. - Complexity hiding. Existing user-space multimedia frameworks all
  57. have existing enums/structures for each compressed format. This new
  58. API assumes the existence of a platform-specific compatibility layer
  59. to expose, translate and make use of the capabilities of the audio
  60. DSP, eg. Android HAL or PulseAudio sinks. By construction, regular
  61. applications are not supposed to make use of this API.
  62. Design
  63. ======
  64. The new API shares a number of concepts with the PCM API for flow
  65. control. Start, pause, resume, drain and stop commands have the same
  66. semantics no matter what the content is.
  67. The concept of memory ring buffer divided in a set of fragments is
  68. borrowed from the ALSA PCM API. However, only sizes in bytes can be
  69. specified.
  70. Seeks/trick modes are assumed to be handled by the host.
  71. The notion of rewinds/forwards is not supported. Data committed to the
  72. ring buffer cannot be invalidated, except when dropping all buffers.
  73. The Compressed Data API does not make any assumptions on how the data
  74. is transmitted to the audio DSP. DMA transfers from main memory to an
  75. embedded audio cluster or to a SPI interface for external DSPs are
  76. possible. As in the ALSA PCM case, a core set of routines is exposed;
  77. each driver implementer will have to write support for a set of
  78. mandatory routines and possibly make use of optional ones.
  79. The main additions are
  80. get_caps
  81. This routine returns the list of audio formats supported. Querying the
  82. codecs on a capture stream will return encoders, decoders will be
  83. listed for playback streams.
  84. get_codec_caps
  85. For each codec, this routine returns a list of
  86. capabilities. The intent is to make sure all the capabilities
  87. correspond to valid settings, and to minimize the risks of
  88. configuration failures. For example, for a complex codec such as AAC,
  89. the number of channels supported may depend on a specific profile. If
  90. the capabilities were exposed with a single descriptor, it may happen
  91. that a specific combination of profiles/channels/formats may not be
  92. supported. Likewise, embedded DSPs have limited memory and cpu cycles,
  93. it is likely that some implementations make the list of capabilities
  94. dynamic and dependent on existing workloads. In addition to codec
  95. settings, this routine returns the minimum buffer size handled by the
  96. implementation. This information can be a function of the DMA buffer
  97. sizes, the number of bytes required to synchronize, etc, and can be
  98. used by userspace to define how much needs to be written in the ring
  99. buffer before playback can start.
  100. set_params
  101. This routine sets the configuration chosen for a specific codec. The
  102. most important field in the parameters is the codec type; in most
  103. cases decoders will ignore other fields, while encoders will strictly
  104. comply to the settings
  105. get_params
  106. This routines returns the actual settings used by the DSP. Changes to
  107. the settings should remain the exception.
  108. get_timestamp
  109. The timestamp becomes a multiple field structure. It lists the number
  110. of bytes transferred, the number of samples processed and the number
  111. of samples rendered/grabbed. All these values can be used to determine
  112. the average bitrate, figure out if the ring buffer needs to be
  113. refilled or the delay due to decoding/encoding/io on the DSP.
  114. Note that the list of codecs/profiles/modes was derived from the
  115. OpenMAX AL specification instead of reinventing the wheel.
  116. Modifications include:
  117. - Addition of FLAC and IEC formats
  118. - Merge of encoder/decoder capabilities
  119. - Profiles/modes listed as bitmasks to make descriptors more compact
  120. - Addition of set_params for decoders (missing in OpenMAX AL)
  121. - Addition of AMR/AMR-WB encoding modes (missing in OpenMAX AL)
  122. - Addition of format information for WMA
  123. - Addition of encoding options when required (derived from OpenMAX IL)
  124. - Addition of rateControlSupported (missing in OpenMAX AL)
  125. State Machine
  126. =============
  127. The compressed audio stream state machine is described below ::
  128. +----------+
  129. | |
  130. | OPEN |
  131. | |
  132. +----------+
  133. |
  134. |
  135. | compr_set_params()
  136. |
  137. v
  138. compr_free() +----------+
  139. +------------------------------------| |
  140. | | SETUP |
  141. | +-------------------------| |<-------------------------+
  142. | | compr_write() +----------+ |
  143. | | ^ |
  144. | | | compr_drain_notify() |
  145. | | | or |
  146. | | | compr_stop() |
  147. | | | |
  148. | | +----------+ |
  149. | | | | |
  150. | | | DRAIN | |
  151. | | | | |
  152. | | +----------+ |
  153. | | ^ |
  154. | | | |
  155. | | | compr_drain() |
  156. | | | |
  157. | v | |
  158. | +----------+ +----------+ |
  159. | | | compr_start() | | compr_stop() |
  160. | | PREPARE |------------------->| RUNNING |--------------------------+
  161. | | | | | |
  162. | +----------+ +----------+ |
  163. | | | ^ |
  164. | |compr_free() | | |
  165. | | compr_pause() | | compr_resume() |
  166. | | | | |
  167. | v v | |
  168. | +----------+ +----------+ |
  169. | | | | | compr_stop() |
  170. +--->| FREE | | PAUSE |---------------------------+
  171. | | | |
  172. +----------+ +----------+
  173. Gapless Playback
  174. ================
  175. When playing thru an album, the decoders have the ability to skip the encoder
  176. delay and padding and directly move from one track content to another. The end
  177. user can perceive this as gapless playback as we don't have silence while
  178. switching from one track to another
  179. Also, there might be low-intensity noises due to encoding. Perfect gapless is
  180. difficult to reach with all types of compressed data, but works fine with most
  181. music content. The decoder needs to know the encoder delay and encoder padding.
  182. So we need to pass this to DSP. This metadata is extracted from ID3/MP4 headers
  183. and are not present by default in the bitstream, hence the need for a new
  184. interface to pass this information to the DSP. Also DSP and userspace needs to
  185. switch from one track to another and start using data for second track.
  186. The main additions are:
  187. set_metadata
  188. This routine sets the encoder delay and encoder padding. This can be used by
  189. decoder to strip the silence. This needs to be set before the data in the track
  190. is written.
  191. set_next_track
  192. This routine tells DSP that metadata and write operation sent after this would
  193. correspond to subsequent track
  194. partial drain
  195. This is called when end of file is reached. The userspace can inform DSP that
  196. EOF is reached and now DSP can start skipping padding delay. Also next write
  197. data would belong to next track
  198. Sequence flow for gapless would be:
  199. - Open
  200. - Get caps / codec caps
  201. - Set params
  202. - Set metadata of the first track
  203. - Fill data of the first track
  204. - Trigger start
  205. - User-space finished sending all,
  206. - Indicate next track data by sending set_next_track
  207. - Set metadata of the next track
  208. - then call partial_drain to flush most of buffer in DSP
  209. - Fill data of the next track
  210. - DSP switches to second track
  211. (note: order for partial_drain and write for next track can be reversed as well)
  212. Gapless Playback SM
  213. ===================
  214. For Gapless, we move from running state to partial drain and back, along
  215. with setting of meta_data and signalling for next track ::
  216. +----------+
  217. compr_drain_notify() | |
  218. +------------------------>| RUNNING |
  219. | | |
  220. | +----------+
  221. | |
  222. | |
  223. | | compr_next_track()
  224. | |
  225. | V
  226. | +----------+
  227. | | |
  228. | |NEXT_TRACK|
  229. | | |
  230. | +----------+
  231. | |
  232. | |
  233. | | compr_partial_drain()
  234. | |
  235. | V
  236. | +----------+
  237. | | |
  238. +------------------------ | PARTIAL_ |
  239. | DRAIN |
  240. +----------+
  241. Not supported
  242. =============
  243. - Support for VoIP/circuit-switched calls is not the target of this
  244. API. Support for dynamic bit-rate changes would require a tight
  245. coupling between the DSP and the host stack, limiting power savings.
  246. - Packet-loss concealment is not supported. This would require an
  247. additional interface to let the decoder synthesize data when frames
  248. are lost during transmission. This may be added in the future.
  249. - Volume control/routing is not handled by this API. Devices exposing a
  250. compressed data interface will be considered as regular ALSA devices;
  251. volume changes and routing information will be provided with regular
  252. ALSA kcontrols.
  253. - Embedded audio effects. Such effects should be enabled in the same
  254. manner, no matter if the input was PCM or compressed.
  255. - multichannel IEC encoding. Unclear if this is required.
  256. - Encoding/decoding acceleration is not supported as mentioned
  257. above. It is possible to route the output of a decoder to a capture
  258. stream, or even implement transcoding capabilities. This routing
  259. would be enabled with ALSA kcontrols.
  260. - Audio policy/resource management. This API does not provide any
  261. hooks to query the utilization of the audio DSP, nor any preemption
  262. mechanisms.
  263. - No notion of underrun/overrun. Since the bytes written are compressed
  264. in nature and data written/read doesn't translate directly to
  265. rendered output in time, this does not deal with underrun/overrun and
  266. maybe dealt in user-library
  267. Credits
  268. =======
  269. - Mark Brown and Liam Girdwood for discussions on the need for this API
  270. - Harsha Priya for her work on intel_sst compressed API
  271. - Rakesh Ughreja for valuable feedback
  272. - Sing Nallasellan, Sikkandar Madar and Prasanna Samaga for
  273. demonstrating and quantifying the benefits of audio offload on a
  274. real platform.