RSS
This describes the sysv_ipc module which gives Python access to System V inter-process semaphores, shared memory and message queues on most (all?) *nix flavors. Examples include macOS/OS X, Linux, FreeBSD, OpenSolaris 2008.11, HPUX, and AIX. It might also work under Windows with a library like Cygwin.
It works with Python 2.7 and 3.x. It's released under a BSD license.
You can download sysv_ipc version 1.0.0 ([md5 sum], [sha1 sum], [sha256 sum]) which contains the source code, setup.py, installation instructions and sample code. You can read about all of the changes in this version.
You might also want to read about some known bugs.
You might be interested in the very similar module posix_ipc which provides Python access to POSIX IPC primitives. POSIX IPC is a little easier to use than SysV IPC, but not all operating systems support it completely.
Jump to semaphores, shared memory, or message queues.
This method is useful only under fairly unusual circumstances. You probably don't need it.
When passed to an IPC object's constructor, IPC_CREAT indicates that you want to create a new object or open an existing one. If you want the call to fail if an object with that key already exists, specify the IPC_EXCL flag, too.
Keys randomly generated by this module are in the range 1 ≤ key ≤ SHRT_MAX. That's type-safe unless your OS has a very bizarre definition of key_t.
In addition to standard Python errors (e.g. ValueError), this module raises custom errors. These errors cover situations specific to IPC.
This is a handle to a semaphore.
key must be None, IPC_PRIVATE or an integer > KEY_MIN and ≤ KEY_MAX. If the key is None, the module chooses a random unused key.
The flags specify whether you want to create a new semaphore or open an existing one.
When opening an existing semaphore, mode is ignored.
The timeout (which can be a float) specifies how many seconds this call should wait, if at all.
The semantics of the timeout changed a little in version 0.3.
When the call returns, the semaphore's value decreases by delta (or more precisely, abs(delta)) which defaults to 1.
On platforms that don't support the semtimedop() API call, all timeouts (including zero) are treated as infinite. The call will not return until its wait condition is satisfied.
Most platforms provide semtimedop(). macOS is a notable exception. The module's Boolean constant SEMAPHORE_TIMEOUT_SUPPORTED is True on platforms that support semtimedop().
The semaphore's value increases by delta (or more precisely, abs(delta)) which defaults to 1.
"P" stands for prolaag or probeer te verlagen (try to decrease), the original name given by Edsger Dijkstra.
"V" stands for verhoog (increase), the original name given by Edsger Dijkstra.
Timeout has the same meaning as described in .acquire().
As far as I can tell, the effect of deleting a semaphore that other processes are still using is OS-dependent. Check your system's man pages for semctl(IPC_RMID).
When True, operations that change the semaphore's value will be undone (reversed) when the process exits. Note that when a process exits, an undo operation may imply that a semaphore's value should become negative or exceed its maximum. Behavior in this case is system-dependent, which means that using this flag can make your code non-portable.
When False, these calls will not block but will instead raise an error if they are unable to return immediately.
Tip: the following Python code will display
the mode in octal:
print int(str(my_sem.mode), 8)
Linux and macOS also set this when the semaphore's value is changed, although doing so disagrees with the POSIX specification. See Linux kernel bug 112271.
These semaphores provide __enter__() and __exit__() methods so they can be used in context managers. For instance --
with sysv_ipc.Semaphore(name) as sem:
# Do something...
Entering the context acquires the semaphore, exiting the context releases the semaphore. See demo4/child.py for a complete example.
This is a handle to a shared memory segment.
In addition to the methods and attributes described below, SharedMemory supports Python's buffer protocol, which means you can create bytearray and memoryview objects based on a SharedMemory segment. See demos/buffer_protocol for an example.
key must be None, IPC_PRIVATE or an integer > 0 and ≤ KEY_MAX. If the key is None, the module chooses a random unused key.
The flags specify whether you want to create a new shared memory segment or open an existing one.
When both IPC_CREX is specified and the caller has write permission, each byte in the new memory segment will be initialized to the value of init_character.
The value of size depends on whether one is opening an existing segment or creating a new one.
This module supplies a default size of PAGE_SIZE when IPC_CREX is specified and 0 otherwise.
The address parameter allows one to specify (as a Python long) a memory address at which to attach the segment. Passing None (the default) is equivalent to passing NULL to shmat(). See that function's man page for details.
The flags are mostly only relevant if one specifies a specific address. One exception is the flag SHM_RDONLY which, surprisingly, attaches the segment read-only.
Note that on some (and perhaps all) platforms, each call to .attach() increments the system's "attached" count. Thus, if each call to .attach() isn't paired with a call to .detach(), the system's "attached" count for the shared memory segment will not go to zero when the process exits. As a result, the shared memory segment may not disappear even when its creator calls .remove() and exits.
If byte_count is zero (the default) the entire buffer is returned.
This method will never attempt to read past the end of the shared memory segment, even when offset + byte_count exceeds the memory segment's size. In that case, the bytes from offset to the end of the segment are returned.
If the offset + data would write outside of the segment, this function raises ValueError.
The bytes may contain embedded NULL bytes ('\0').
Tip: the following Python code will display
the mode in octal:
print int(str(my_mem.mode), 8)
This is a handle to a FIFO message queue.
key must be None, IPC_PRIVATE or an integer > 0 and ≤ KEY_MAX. If the key is None, the module chooses a random unused key.
The flags specify whether you want to create a new queue or open an existing one.
The max_message_size can be increased from the default, but be aware of the issues discussed in Message Queue Limits.
The message should be a bytes object (a.k.a. str in Python 2) and can contain embedded NULLs (ASCII 0x00). Passing a Unicode object may work, but doing so is unsupported and may be explicitly deprecated in a future version.
The block flag specifies whether or not the call should wait if the message can't be sent (if, for example, the queue is full). When block is False, the call will raise a BusyError if the message can't be sent immediately.
The type is associated with the message and is relevant when calling receive(). It must be > 0.
The block flag specifies whether or not the call should wait if there's no messages of the specified type to retrieve. When block is False, the call will raise a BusyError if a message can't be received immediately.
The type permits some control over which messages are retrieved.
Tip: the following Python code will display
the mode in octal:
print int(str(my_mem.mode), 8)
This module comes with four sets of demonstration code in the directory demos.
Most System V IPC sample code recommends ftok() for generating an integer key that's more-or-less random. It does not, however, guarantee that the key it generates is unused. If ftok() gives your application a key that some other application is already using, your app is in trouble unless it has a reliable second mechanism for generating a key. And if that's the case, why not just abandon ftok() and use the second mechanism exclusively?
This is the weakness of ftok() -- it isn't guaranteed to give you what you want. The BSD man page for ftok says it is "quite possible for the routine to return duplicate keys". The term "quite possible" isn't quantified, but suppose it means one-tenth of one percent. Who wants to have 1-in-1000 odds of a catastrophic failure in their program, or even 1-in-10000?
This module obviates the need for ftok() by generating random keys for you. If your application can't use sysv_ipc's automatically generated keys because it needs to know the key in advance, hardcoding a random number like 123456 in your app might be no worse than using ftok() and has the advantage of not hiding its limitations.
This module provides ftok() in case you want to experiment with it. However, to emphasize its weakness, this version of ftok() raises a warning with every call unless you explicitly pass a flag to silence it.
This package also provides ftok_experiment.py so that you can observe how often ftok() generates duplicate keys on your system.
When a System V sempahore is created at the C API level, the OS is not required to initialize the semaphore's value. (This per the SUSv3 standard for semget().) Some (most? all?) operating systems initialize it to zero, but this behavior is non-standard and therefore can't be relied upon.
If sempahore creation happens in an predictable, orderly fashion, this isn't a problem. But a race condition arises when multiple processes vie to create/open the same semaphore. The problem lies in the fact that when an application calls semget() with only the IPC_CREAT flag, the caller can't tell whether or not he has created a new semaphore or opened an existing one. This makes it difficult to create reliable code without using IPC_EXCL. W. Richard Stevens' Unix Network Programming Volume 2 calls this "a fatal flaw in the design of System V semaphores" (p 284).
For instance, imagine processes P1 and P2. They're executing the same code, and that code intends to share a binary semaphore. Consider the following sequence of events at the startup of P1 and P2 –
W. Richard Stevens' solution for this race condition is to check the value of sem_otime (an element in the semid_ds struct that's populated on the call to semctl(IPC_STAT) and which is exposed to Python by this module) which is initialized to zero when the semaphore is created and otherwise holds the time of the last call to semop() (which is called by P()/acquire(), V()/release(), and Z()).
In Python, each process would run something like this:
try:
sem = sysv_ipc.Semaphore(42, sysv_ipc.IPC_CREX)
except sysv_ipc.ExistentialError:
# One of my peers created the semaphore already
sem = sysv_ipc.Semaphore(42)
# Waiting for that peer to do the first acquire or release
while not sem.o_time:
time.sleep(.1)
else:
# Initializing sem.o_time to nonzero value
sem.release()
# Now the semaphore is safe to use.
With shared memory, using the IPC_CREAT flag without IPC_EXCL is problematic unless you know the size of the segment you're potentially opening.
Why? Because when creating a new segment, many (most? all?) operating systems demand a non-zero size. However, when opening an existing segment, zero is the only guaranteed safe value (again, assuming one doesn't know the size of the segment in advance). Since IPC_CREAT can open or create a segment, there's no safe value for the size under this circumstance.
As a (sort of) side note, the SUSv3 specification for shmget() says only that the size of a new segment must not be less than "the system-imposed minimum". I gather that at one time, some systems set the minimum at zero despite the fact that it doesn't make much sense to create a zero-length shared memory segment. I think most modern systems do the sensible thing and insist on a minimum length of 1.
Python programmers can usually remain blissfully ignorant of memory allocation issues. Unfortunately, a combination of factors makes them relevant when dealing with System V message queues.
Some implementations impose extremely stingy limits. For instance, many BSDish systems (macOS, FreeBSD, NetBSD, and OpenBSD) limit queues to 2048 bytes. Note that that's the total queue size, not the message size. Two 1k messages would fill the queue.
Those limits can be very difficult to change. At best, only privileged processes can increase the limit. At worst, the limit is a kernel parameter and requires a kernel change via a tunable or a recompile.
This module can't figure out what the limits are, so it can't cushion them or even report them to you. On some systems the limits are expressed in header files, on others they're available through kernel interfaces (like FreeBSD's sysctl). Under macOS and to some extent OpenSolaris I can't figure out where they're defined and what I report here is the result of experimentation and educated guesses formed by Googling.
The good news is that this module will still behave as advertised no matter what these limits are. Nevertheless you might be surprised when a call to .send() get stuck because a queue is full even though you've only put 2048 bytes of messages in it.
Here are the limits I've been able to find under my test operating systems, ordered from best (most generous) to worst (most stingy). This information was current as of 2009 when I wrote the message queue code. It's getting pretty stale now. I hope the situation has improved over the 2009 numbers I describe below.
Under OpenSolaris 2008.05 each queue's maximum size defaults to 64k. A privileged process (e.g. root) can change this through the max_size attribute of a sysv_ipc.MessageQueue object. I was able to increase it to 16M and successfully sent sixteen 1M messages to the queue.
Under Ubuntu 8.04 (and perhaps other Linuxes) each queue's maximum size defaults to 16k. As with OpenSolaris, I was able to increase this to 16M, but only for a privileged process.
Under FreeBSD 7 and I think NetBSD and OpenBSD, each queue's maximum size defaults to 2048 bytes. Furthermore, one can (as root) set max_size to something larger and FreeBSD doesn't complain, but it also ignores the change.
macOS is the worst of the lot. Each queue is limited to 2048 bytes and macOS silently ignores attempts to increase this (just like FreeBSD). To add insult to injury, there appears to be no way to increase this limit short of recompiling the kernel. I'm guessing at this based on the Darwin message queue limits.
If you want to search for these limits on your operating system, the key constants are MSGSEG, MSGSSZ, MSGTQL, MSGMNB, MSGMNI and MSGMAX. Under BSD, sysctl kern.ipc should tell you what you need to know and may allow you to change these parameters.
Semaphores and especially shared memory are a little different from most Python objects and therefore require a little more care on the part of the programmer. When a program creates a semaphore or shared memory object, it creates something that resides outside of its own process, just like a file on a hard drive. It won't go away when your process ends unless you explicitly remove it.
In short, remember to clean up after yourself.
The sysv_ipc module is just a wrapper around your system's API. If your system's implementation has quirks, the man pages for semget, semctl, semop shmget, shmat, shmdt and shmctl will probably cover them.
Many systems (although not some older versions of OS X) come with ipcs and ipcrm. The former shows existing shared memory, semaphores and message queues on your system and the latter allows you to remove them.
For Pythonistas –
Bugs? My code never has bugs! There are, however, some suboptimal anomalies...
These are features that may or may not be added depending on technical difficulty, user interest and so forth.
I don't plan on adding support for semaphore sets.