Device Power Management (C) 2010 Rafael J. Wysocki , Novell Inc. Most of the code in Linux is device drivers, so most of the Linux power management code is also driver-specific. Most drivers will do very little; others, especially for platforms with small batteries (like cell phones), will do a lot. This writeup gives an overview of how drivers interact with system-wide power management goals, emphasizing the models and interfaces that are shared by everything that hooks up to the driver model core. Read it as background for the domain-specific work you'd do with any specific driver. Two Models for Device Power Management ====================================== Drivers will use one or both of these models to put devices into low-power states: System Sleep model: Drivers can enter low power states as part of entering system-wide low-power states like "suspend-to-ram", or (mostly for systems with disks) "hibernate" (suspend-to-disk). This is something that device, bus, and class drivers collaborate on by implementing various role-specific suspend and resume methods to cleanly power down hardware and software subsystems, then reactivate them without loss of data. Some drivers can manage hardware wakeup events, which make the system leave that low-power state. This feature may be enabled or disabled using the relevant /sys/devices/.../power/wakeup file (for Ethernet drivers the ioctl interface used by ethtool may also be used for this purpose); enabling it may cost some power usage, but let the whole system enter low power states more often. Runtime Power Management model: Devices may also be put into low power states while the system is running, independently of other power management activity in principle. However, devices are not generally independent of each other (for example, parent device cannot be suspended unless all of its child devices have been suspended). Moreover, depending on the bus type the device is on, it may be necessary to carry out some bus-specific operations on the device for this purpose. Also, devices put into low power states at run time may require special handling during system-wide power transitions, like suspend to RAM. For these reasons not only the device driver itself, but also the appropriate subsystem (bus type, device type or device class) driver and the PM core are involved in the runtime power management of devices. Like in the system sleep power management case, they need to collaborate by implementing various role-specific suspend and resume methods, so that the hardware is cleanly powered down and reactivated without data or service loss. There's not a lot to be said about those low power states except that they are very system-specific, and often device-specific. Also, that if enough devices have been put into low power states (at "run time"), the effect may be very similar to entering some system-wide low-power state (system sleep) ... and that synergies exist, so that several drivers using runtime PM might put the system into a state where even deeper power saving options are available. Most suspended devices will have quiesced all I/O: no more DMA or IRQs, no more data read or written, and requests from upstream drivers are no longer accepted. A given bus or platform may have different requirements though. Examples of hardware wakeup events include an alarm from a real time clock, network wake-on-LAN packets, keyboard or mouse activity, and media insertion or removal (for PCMCIA, MMC/SD, USB, and so on). Interfaces for Entering System Sleep States =========================================== There are programming interfaces provided for subsystem (bus type, device type, device class) and device drivers in order to allow them to participate in the power management of devices they are concerned with. They cover the system sleep power management as well as the runtime power management of devices. Device Power Management Operations ---------------------------------- Device power management operations, at the subsystem level as well as at the device driver level, are implemented by defining and populating objects of type struct dev_pm_ops: struct dev_pm_ops { int (*prepare)(struct device *dev); void (*complete)(struct device *dev); int (*suspend)(struct device *dev); int (*resume)(struct device *dev); int (*freeze)(struct device *dev); int (*thaw)(struct device *dev); int (*poweroff)(struct device *dev); int (*restore)(struct device *dev); int (*suspend_noirq)(struct device *dev); int (*resume_noirq)(struct device *dev); int (*freeze_noirq)(struct device *dev); int (*thaw_noirq)(struct device *dev); int (*poweroff_noirq)(struct device *dev); int (*restore_noirq)(struct device *dev); int (*runtime_suspend)(struct device *dev); int (*runtime_resume)(struct device *dev); int (*runtime_idle)(struct device *dev); }; This structure is defined in include/linux/pm.h and the methods included in it are also described in that file. Their roles will be explained in what follows. For now, it should be sufficient to remember that the last three of them are specific to runtime power management, while the remaining ones are used during system-wide power transitions. There also is an "old" or "legacy", deprecated way of implementing power management operations available at least for some subsystems. This approach does not use struct dev_pm_ops objects and it only is suitable for implementing system sleep power management methods. Therefore it is not described in this document, so please refer directly to the source code for more information about it. Subsystem-Level Methods ----------------------- The core methods to suspend and resume devices reside in struct dev_pm_ops pointed to by the pm member of struct bus_type, struct device_type and struct class. They are mostly of interest to the people writing infrastructure for buses, like PCI or USB, or device type and device class drivers. Bus drivers implement these methods as appropriate for the hardware and the drivers using it; PCI works differently from USB, and so on. Not many people write subsystem-level drivers; most driver code is a "device driver" that builds on top of bus-specific framework code. For more information on these driver calls, see the description later; they are called in phases for every device, respecting the parent-child sequencing in the driver model tree. /sys/devices/.../power/wakeup files ----------------------------------- All devices in the driver model have two flags to control handling of wakeup events, which are hardware signals that can force the device and/or system out of a low power state. These are initialized by bus or device driver code using device_init_wakeup(). The "can_wakeup" flag just records whether the device (and its driver) can physically support wakeup events. When that flag is clear, the sysfs "wakeup" file is empty, and device_may_wakeup() returns false. For devices that can issue wakeup events, a separate flag controls whether that device should try to use its wakeup mechanism. The initial value of device_may_wakeup() will be false for the majority of devices, except for power buttons, keyboards, and Ethernet adapters whose WoL (wake-on-LAN) feature has been set up with ethtool. Thus in the majority of cases the device's "wakeup" file will initially hold the value "disabled". Userspace can change that to "enabled", so that device_may_wakeup() returns true, or change it back to "disabled", so that it returns false again. /sys/devices/.../power/control files ------------------------------------ All devices in the driver model have a flag to control the desired behavior of its driver with respect to runtime power management. This flag, called runtime_auto, is initialized by the bus type (or generally subsystem) code using pm_runtime_allow() or pm_runtime_forbid(), depending on whether or not the driver is supposed to power manage the device at run time by default, respectively. This setting may be adjusted by user space by writing either "on" or "auto" to the device's "control" file. If "auto" is written, the device's runtime_auto flag will be set and the driver will be allowed to power manage the device if capable of doing that. If "on" is written, the driver is not allowed to power manage the device which in turn is supposed to remain in the full power state at run time. User space can check the current value of the runtime_auto flag by reading from the device's "control" file. The device's runtime_auto flag has no effect on the handling of system-wide power transitions by its driver. In particular, the device can (and in the majority of cases should and will) be put into a low power state during a system-wide transition to a sleep state (like "suspend-to-RAM") even though its runtime_auto flag is unset (in which case its "control" file contains "on"). For more information about the runtime power management framework for devices refer to Documentation/power/runtime_pm.txt. Calling Drivers to Enter System Sleep States ============================================ When the system goes into a sleep state, each device's driver is asked to suspend the device by putting it into state compatible with the target system state. That's usually some version of "off", but the details are system-specific. Also, wakeup-enabled devices will usually stay partly functional in order to wake the system. When the system leaves that low power state, the device's driver is asked to resume it. The suspend and resume operations always go together, and both are multi-phase operations. For simple drivers, suspend might quiesce the device using the class code and then turn its hardware as "off" as possible with late_suspend. The matching resume calls would then completely reinitialize the hardware before reactivating its class I/O queues. More power-aware drivers might prepare the devices for triggering system wakeup events. Call Sequence Guarantees ------------------------ To ensure that bridges and similar links needing to talk to a device are available when the device is suspended or resumed, the device tree is walked in a bottom-up order to suspend devices. A top-down order is used to resume those devices. The ordering of the device tree is defined by the order in which devices get registered: a child can never be registered, probed or resumed before its parent; and can't be removed or suspended after that parent. The policy is that the device tree should match hardware bus topology. (Or at least the control bus, for devices which use multiple busses.) In particular, this means that a device registration may fail if the parent of the device is suspending (i.e. has been chosen by the PM core as the next device to suspend) or has already suspended, as well as after all of the other devices have been suspended. Device drivers must be prepared to cope with such situations. Suspending Devices ------------------ Suspending a given device is done in several phases. Suspending the system always includes every phase, executing calls for every device before the next phase begins. Not all busses or classes support all these callbacks; and not all drivers use all the callbacks. Generally, different callbacks are used depending on whether the system is going to the standby or memory sleep state ("suspend-to-RAM") or it is going to be hibernated ("suspend-to-disk"). If the system goes to the standby or memory sleep state the phases are seen by driver notifications issued in this order: 1 bus->pm.prepare(dev) is called after tasks are frozen and it is supposed to call the device driver's ->pm.prepare() method. The purpose of this method is mainly to prevent new children of the device from being registered after it has returned. It also may be used to generally prepare the device for the upcoming system transition, but it should not put the device into a low power state. 2 class->pm.suspend(dev) is called if dev is associated with a class that has such a method. It may invoke the device driver's ->pm.suspend() method, unless type->pm.suspend(dev) or bus->pm.suspend() does that. 3 type->pm.suspend(dev) is called if dev is associated with a device type that has such a method. It may invoke the device driver's ->pm.suspend() method, unless class->pm.suspend(dev) or bus->pm.suspend() does that. 4 bus->pm.suspend(dev) is called, if implemented. It usually calls the device driver's ->pm.suspend() method. This call should generally quiesce the device so that it doesn't do any I/O after the call has returned. It also may save the device registers and put it into the appropriate low power state, depending on the bus type the device is on. 5 bus->pm.suspend_noirq(dev) is called, if implemented. It may call the device driver's ->pm.suspend_noirq() method, depending on the bus type in question. This method is invoked after device interrupts have been suspended, which means that the driver's interrupt handler will not be called while it is running. It should save the values of the device's registers that weren't saved previously and finally put the device into the appropriate low power state. The majority of subsystems and device drivers need not implement this method. However, bus types allowing devices to share interrupt vectors, like PCI, generally need to use it to prevent interrupt handling issues from happening during suspend. At the end of those phases, drivers should normally have stopped all I/O transactions (DMA, IRQs), saved enough state that they can re-initialize or restore previous state (as needed by the hardware), and placed the device into a low-power state. On many platforms they will also use gate off one or more clock sources; sometimes they will also switch off power supplies, or reduce voltages. [Drivers supporting runtime PM may already have performed some or all of the steps needed to prepare for the upcoming system state transition.] If device_may_wakeup(dev) returns true, the device should be prepared for generating hardware wakeup signals when the system is in the sleep state to trigger a system wakeup event. For example, enable_irq_wake() might identify GPIO signals hooked up to a switch or other external hardware, and pci_enable_wake() does something similar for the PCI PME signal. If a driver (or subsystem) fails it suspend method, the system won't enter the desired low power state; it will resume all the devices it's suspended so far. Hibernation Phases ------------------ Hibernating the system is more complicated than putting it into the standby or memory sleep state, because it involves creating a system image and saving it. Therefore there are more phases of hibernation and special device PM methods are used in this case. First, it is necessary to prepare the system for creating a hibernation image. This is similar to putting the system into the standby or memory sleep state, although it generally doesn't require that devices be put into low power states (that is even not desirable at this point). Driver notifications are then issued in the following order: 1 bus->pm.prepare(dev) is called after tasks have been frozen and enough memory has been freed. 2 class->pm.freeze(dev) is called if implemented. It may invoke the device driver's ->pm.freeze() method, unless type->pm.freeze(dev) or bus->pm.freeze() does that. 3 type->pm.freeze(dev) is called if implemented. It may invoke the device driver's ->pm.suspend() method, unless class->pm.freeze(dev) or bus->pm.freeze() does that. 4 bus->pm.freeze(dev) is called, if implemented. It usually calls the device driver's ->pm.freeze() method. 5 bus->pm.freeze_noirq(dev) is called, if implemented. It may call the device driver's ->pm.freeze_noirq() method, depending on the bus type in question. The difference between ->pm.freeze() and the corresponding ->pm.suspend() (and similarly for the "noirq" variants) is that the former should avoid preparing devices to trigger system wakeup events and putting devices into low power states, although they generally have to save the values of device registers so that it's possible to restore them during system resume. Second, after the system image has been created, the functionality of devices has to be restored so that the image can be saved. That is similar to resuming devices after the system has been woken up from the standby or memory sleep state, which is described below, and causes the following device notifications to be issued: 1 bus->pm.thaw_noirq(dev), if implemented; may call the device driver's ->pm.thaw_noirq() method, depending on the bus type in question. 2 bus->pm.thaw(dev), if implemented; usually calls the device driver's ->pm.thaw() method. 3 type->pm.thaw(dev), if implemented; may call the device driver's ->pm.thaw() method if not called by the bus type or class. 4 class->pm.thaw(dev), if implemented; may call the device driver's ->pm.thaw() method if not called by the bus type or device type. 5 bus->pm.complete(dev), if implemented; may call the device driver's ->pm.complete() method. Generally, the role of the ->pm.thaw() methods (including the "noirq" variants) is to bring the device back to the fully functional state, so that it may be used for saving the image, if necessary. The role of bus->pm.complete() is to reverse whatever bus->pm.prepare() did (likewise for the analogous device driver callbacks). After the image has been saved, the devices need to be prepared for putting the system into the low power state. That is analogous to suspending them before putting the system into the standby or memory sleep state and involves the following device notifications: 1 bus->pm.prepare(dev). 2 class->pm.poweroff(dev), if implemented; may invoke the device driver's ->pm.poweroff() method if not called by the bus type or device type. 3 type->pm.poweroff(dev), if implemented; may invoke the device driver's ->pm.poweroff() method if not called by the bus type or device class. 4 bus->pm.poweroff(dev), if implemented; usually calls the device driver's ->pm.poweroff() method (if not called by the device class or type). 5 bus->pm.poweroff_noirq(dev), if implemented; may call the device driver's ->pm.poweroff_noirq() method, depending on the bus type in question. The difference between ->pm.poweroff() and the corresponding ->pm.suspend() (and analogously for the "noirq" variants) is that the former need not save the device's registers. Still, they should prepare the device for triggering system wakeup events if necessary and finally put it into the appropriate low power state. Device Low Power (suspend) States --------------------------------- Device low-power states aren't standard. One device might only handle "on" and "off, while another might support a dozen different versions of "on" (how many engines are active?), plus a state that gets back to "on" faster than from a full "off". Some busses define rules about what different suspend states mean. PCI gives one example: after the suspend sequence completes, a non-legacy PCI device may not perform DMA or issue IRQs, and any wakeup events it issues would be issued through the PME# bus signal. Plus, there are several PCI-standard device states, some of which are optional. In contrast, integrated system-on-chip processors often use IRQs as the wakeup event sources (so drivers would call enable_irq_wake) and might be able to treat DMA completion as a wakeup event (sometimes DMA can stay active too, it'd only be the CPU and some peripherals that sleep). Some details here may be platform-specific. Systems may have devices that can be fully active in certain sleep states, such as an LCD display that's refreshed using DMA while most of the system is sleeping lightly ... and its frame buffer might even be updated by a DSP or other non-Linux CPU while the Linux control processor stays idle. Moreover, the specific actions taken may depend on the target system state. One target system state might allow a given device to be very operational; another might require a hard shut down with re-initialization on resume. And two different target systems might use the same device in different ways; the aforementioned LCD might be active in one product's "standby", but a different product using the same SOC might work differently. Resuming Devices ---------------- Resuming is done in multiple phases, much like suspending, with all devices processing each phase's calls before the next phase begins. Again, however, different callbacks are used depending on whether the system is waking up from the standby or memory sleep state ("suspend-to-RAM") or from hibernation ("suspend-to-disk"). If the system is waking up from the standby or memory sleep state, the phases are seen by driver notifications issued in this order: 1 bus->pm.resume_noirq(dev) is called, if implemented. It may call the device driver's ->pm.resume_noirq() method, depending on the bus type in question. The role of this method is to perform actions that need to be performed before device drivers' interrupt handlers are allowed to be invoked. If the given bus type permits devices to share interrupt vectors, like PCI, this method should bring the device and its driver into a state in which the driver can recognize if the device is the source of incoming interrupts, if any, and handle them correctly. For example, the PCI bus type's ->pm.resume_noirq() puts the device into the full power state (D0 in the PCI terminology) and restores the standard configuration registers of the device. Then, it calls the device driver's ->pm.resume_noirq() method to perform device-specific actions needed at this stage of resume. 2 bus->pm.resume(dev) is called, if implemented. It usually calls the device driver's ->pm.resume() method. This call should generally bring the the device back to the working state, so that it can do I/O as requested after the call has returned. However, it may be more convenient to use the device class or device type ->pm.resume() for this purpose, in which case the bus type's ->pm.resume() method need not be implemented at all. 3 type->pm.resume(dev) is called, if implemented. It may invoke the device driver's ->pm.resume() method, unless class->pm.resume(dev) or bus->pm.resume() does that. For devices that are not associated with any bus type or device class this method plays the role of bus->pm.resume(). 4 class->pm.resume(dev) is called, if implemented. It may invoke the device driver's ->pm.resume() method, unless bus->pm.resume(dev) or type->pm.resume() does that. For devices that are not associated with any bus type or device type this method plays the role of bus->pm.resume(). 5 bus->pm.complete(dev) is called, if implemented. It is supposed to invoke the device driver's ->pm.complete() method. The role of this method is to reverse whatever bus->pm.prepare(dev) (or the driver's ->pm.prepare()) did during suspend, if necessary. At the end of those phases, drivers should normally be as functional as they were before suspending: I/O can be performed using DMA and IRQs, and the relevant clocks are gated on. In principle the device need not be "fully on"; it might be in a runtime lowpower/suspend state during suspend and the resume callbacks may try to restore that state, but that need not be desirable from the user's point of view. In fact, there are multiple reasons why it's better to always put devices into the "fully working" state in the system sleep resume callbacks and they are discussed in more detail in Documentation/power/runtime_pm.txt. However, the details here may again be platform-specific. For example, some systems support multiple "run" states, and the mode in effect at the end of resume might not be the one which preceded suspension. That means availability of certain clocks or power supplies changed, which could easily affect how a driver works. Drivers need to be able to handle hardware which has been reset since the suspend methods were called, for example by complete reinitialization. This may be the hardest part, and the one most protected by NDA'd documents and chip errata. It's simplest if the hardware state hasn't changed since the suspend was carried out, but that can't be guaranteed (in fact, it ususally is not the case). Drivers must also be prepared to notice that the device has been removed while the system was powered off, whenever that's physically possible. PCMCIA, MMC, USB, Firewire, SCSI, and even IDE are common examples of busses where common Linux platforms will see such removal. Details of how drivers will notice and handle such removals are currently bus-specific, and often involve a separate thread. Resume From Hibernation ----------------------- Resuming from hibernation is, again, more complicated than resuming from a sleep state in which the contents of main memory are preserved, because it requires a system image to be loaded into memory and the pre-hibernation memory contents to be restored before control can be passed back to the image kernel. In principle, the image might be loaded into memory and the pre-hibernation memory contents might be restored by the boot loader. For this purpose, however, the boot loader would need to know the image kernel's entry point and there's no protocol defined for passing that information to boot loaders. As a workaround, the boot loader loads a fresh instance of the kernel, called the boot kernel, into memory and passes control to it in a usual way. Then, the boot kernel reads the hibernation image, restores the pre-hibernation memory contents and passes control to the image kernel. Thus, in fact, two different kernels are involved in resuming from hibernation and in general they are not only different because they play different roles in this operation. Actually, the boot kernel may be completely different from the image kernel. Not only the configuration of it, but also the version of it may be different. The consequences of this are important to device drivers and their subsystems (bus types, device classes and device types) too. Namely, to be able to load the hibernation image into memory, the boot kernel needs to include at least the subset of device drivers allowing it to access the storage medium containing the image, although it generally doesn't need to include all of the drivers included into the image kernel. After the image has been loaded the devices handled by those drivers need to be prepared for passing control back to the image kernel. This is very similar to the preparation of devices for creating a hibernation image described above. In fact, it is done in the same way, with the help of the ->pm.prepare(), ->pm.freeze() and ->pm.freeze_noirq() callbacks, but only for device drivers included in the boot kernel (whose versions may generally be different from the versions of the analogous drivers from the image kernel). Should the restoration of the pre-hibernation memory contents fail, the boot kernel would carry out the procedure of "thawing" devices described above, using the ->pm.thaw_noirq(), ->pm.thaw(), and ->pm.complete() callbacks provided by subsystems and device drivers. This, however, is a very rare condition. Most often the pre-hibernation memory contents are restored successfully and control is passed to the image kernel that is now responsible for bringing the system back to the working state. To achieve this goal, among other things, the image kernel restores the pre-hibernation functionality of devices. This operation is analogous to the resuming of devices after waking up from the memory sleep state, although it involves different device notifications which are the following: 1 bus->pm.restore_noirq(dev), if implemented; may call the device driver's ->pm.restore_noirq() method, depending on the bus type in question. 2 bus->pm.restore(dev), if implemented; usually calls the device driver's ->pm.restore() method. 3 type->pm.restore(dev), if implemented; may call the device driver's ->pm.restore() method if not called by the bus type or class. 4 class->pm.restore(dev), if implemented; may call the device driver's ->pm.restore() method if not called by the bus type or device type. 5 bus->pm.complete(dev), if implemented; may call the device driver's ->pm.complete() method. The roles of the ->pm.restore_noirq() and ->pm.restore() callbacks are analogous to the roles of the corresponding resume callbacks, but they must assume that the device may have been accessed before by the boot kernel. Consequently, the state of the device before they are called may be different from the state of it right prior to calling the resume callbacks. That difference usually doesn't matter, so the majority of device drivers can set their resume and restore callback pointers to the same routine. Nevertheless, different callback pointers are used in case there is a situation where it actually matters. System Devices -------------- System devices follow a slightly different API, which can be found in include/linux/sysdev.h drivers/base/sys.c System devices will only be suspended with interrupts disabled, and after all other devices have been suspended. On resume, they will be resumed before any other devices, and also with interrupts disabled. That is, when the non-boot CPUs are all offline and IRQs are disabled on the remaining online CPU, then the sysdev_driver.suspend() phase is carried out, and the system enters a sleep state (or hibernation image is created). During resume (or after the image has been created) the sysdev_driver.resume() phase is carried out, IRQs are enabled on the only online CPU, the non-boot CPUs are enabled and that is followed by the "early resume" phase (in which the "noirq" callbacks provided by subsystems and device drivers are invoked). Code to actually enter and exit the system-wide low power state sometimes involves hardware details that are only known to the boot firmware, and may leave a CPU running software (from SRAM or flash memory) that monitors the system and manages its wakeup sequence. Power Management Notifiers -------------------------- As stated in Documentation/power/notifiers.txt, there are some operations that cannot be carried out by the power management callbacks discussed above, because carrying them out at these points would be too late or too early. To handle these cases subsystems and device drivers may register power management notifiers that are called before tasks are frozen and after they have been thawed. Generally speaking, the PM notifiers are suitable for performing actions that either require user space to be available, or at least won't interfere with user space in a wrong way. For details refer to Documentation/power/notifiers.txt. Runtime Power Management ======================== Many devices are able to dynamically power down while the system is still running. This feature is useful for devices that are not being used, and can offer significant power savings on a running system. These devices often support a range of runtime power states, which might use names such as "off", "sleep", "idle", "active", and so on. Those states will in some cases (like PCI) be partially constrained by a bus the device uses, and will usually include hardware states that are also used in system sleep states. Note, however, that a system-wide power transition can be started while some devices are in low power states due to the runtime power management. The system sleep PM callbacks should generally recognize such situations and react to them appropriately, but the recommended actions to be taken in that cases are subsystem-specific. In some cases the decision may be made at the subsystem level while in some other cases the device driver may be left to decide. In some cases it may be desirable to leave a suspended device in that state during system-wide power transition, but in some other cases the device ought to be put back into the full power state, for example to be configured for system wakeup or so that its system wakeup capability can be disabled. That all depends on the hardware and the design of the subsystem and device driver in question. During system-wide resume from a sleep state it's better to put devices into the full power state, as explained in Documentation/power/runtime_pm.txt. Refer to that document for more information regarding this particular issue as well as for information on the device runtime power management framework in general.