The task that issued the I / O request is placed by the supervisor in a state waiting for the ordered operation to complete. When the supervisor receives a message from the completion section that the operation has completed, he puts the task in a ready-to-run state and it continues its work. This situation corresponds to synchronous I / O. Synchronous I / O is standard on most operating systems. To speed up the execution of applications, it has been suggested to use asynchronous I / O if necessary.
The simplest variant of asynchronous output is the so-called buffered output of data to an external device, in which data from the application is transferred not directly to the input / output device, but to a special system buffer. In this case, logically, the output operation for the application is considered completed immediately, and the task may not wait for the end of the actual process of transferring data to the device. The actual output of data from the system buffer is handled by the I / O supervisor. Naturally, the allocation of the buffer from the system memory area is handled by a special system process at the direction of the I / O supervisor. So, for the considered case, the output will be asynchronous if, firstly, the I / O request indicated the need to buffer data, and secondly, if the I / O device allows such asynchronous operations and this is noted in the UCB. You can organize and asynchronous data entry. However, to do this, it is necessary not only to allocate a memory area for temporary storage of data read from the device and to associate the allocated buffer with the task that ordered the operation, but also to split the I / O request itself into two parts (into two requests). The first request specifies the operation to read data, similar to how it is done with synchronous I / O. However, the type (code) of the request is used differently, and the request specifies at least one additional parameter - the name (code) of the system object that the task receives in response to the request and which identifies the allocated buffer. Having received the name of the buffer (we will conventionally call this system object in this way, although in different operating systems other terms are used to designate it, for example, a class), the task continues its work. It is very important to emphasize here that as a result of an asynchronous input request, the task is not placed by the I / O supervisor in a state of waiting for an I / O operation to complete, but remains in an executing or ready-to-execute state. After some time, after executing the necessary code, which was determined by the programmer, the task issues a second request to complete the I / O operation. In this second request to the same device, which, of course, has a different code (or request name), the task specifies the name of the system object (buffer for asynchronous data input) and, if the read operation completes successfully, it immediately receives it from the system buffer. If the data has not yet been fully written from the external device to the system buffer, the I / O supervisor puts the task in the state of waiting for the completion of the I / O operation, and then everything resembles normal synchronous data input.
Typically, asynchronous I / O is provided in most multiprogramming operating systems, especially if the operating system supports multitasking using a threading mechanism. However, if there is no explicit asynchronous I / O, you can implement its ideas yourself by organizing a separate stream for data output.
I / O hardware can be viewed as a collection of hardware processors, which are able to work in parallel with respect to each other, as well as with respect to the central processor (s). On such "processors" the so-called external processes. For example, for an external device (input / output device), an external process can be a set of operations that translate the print head, advance the paper one position, change the ink color, or print some characters. External processes, using input / output equipment, interact both with each other and with ordinary "software" processes running on the central processor. In this case, it is important that the speed of execution of external processes will differ significantly (sometimes, by an order of magnitude or more) from the speed of execution of ordinary (" internal») Processes. For their normal operation, external and internal processes must be synchronized. To smooth out the effect of a strong speed mismatch between internal and external processes, the aforementioned buffering is used. Thus, we can talk about a system of parallel interacting processes (see Chapter 6).
Buffers are a critical resource in relation to internal (software) and external processes, which interact informationally during their parallel development. Through the buffer (buffers), data is either sent from a certain process to an addressable external one (operation of outputting data to an external device), or from an external process is transferred to a certain program process (data reading operation). The introduction of buffering as a means of information interaction raises the problem of managing these system buffers, which is solved by means of the supervisory part of the OS. In this case, the supervisor is entrusted with the tasks of not only allocating and freeing buffers in the system memory area, but also synchronizing processes in accordance with the state of operations for filling or freeing buffers, as well as waiting for them if there are no free buffers available, and the request for input / the output requires buffering. Typically, the I / O supervisor uses the standard synchronization tools adopted in this OS to solve the listed tasks. Therefore, if the OS has developed tools for solving the problems of parallel execution of interacting applications and tasks, then, as a rule, it also implements asynchronous I / O.
Data input / output
Most of the previous articles have been devoted to optimizing computational performance. We've seen many examples of tuning garbage collection, loop parallelization and recursive algorithms, and even tweaked the algorithms to reduce run-time overhead.
For some applications, optimization of the computational aspects provides only marginal performance gains because the bottleneck is I / O operations such as network transfers or disk access. From our experience, we can say that a significant proportion of performance problems are not related to the use of suboptimal algorithms or excessive load on the processor, but to inefficient use of input / output devices. Let's look at two situations where I / O optimization can improve overall performance:
The application can experience severe computational overhead due to inefficient I / O operations that add to the overhead. Worse, the congestion can be so great that it is a limiting factor in making the most of the I / O bandwidth.
An I / O device may not be fully utilized or its capabilities may be wasted due to inefficient programming patterns, such as transferring large amounts of data in small chunks or not using all the bandwidth.
This article describes general I / O concepts and provides recommendations for improving the performance of any type of I / O. These guidelines apply equally to network applications, disk-intensive processes, and even programs accessing non-standard, high-performance hardware devices.
Synchronous and asynchronous I / O
When executed in synchronous mode, Win32 API I / O functions (such as ReadFile, WriteFile, or DeviceloControl) block program execution until the operation completes. Although this model is very easy to use, it is not very effective. In the intervals between the execution of successive I / O requests, the device may be idle, that is, not fully used.
Another problem with synchronous mode is that the thread is wasting time on any concurrent I / O operation. For example, in a server application serving multiple clients at the same time, you might want to create a separate thread of execution for each session. These threads, which are idle most of the time, waste memory and can create situations thread thrashing when multiple threads of execution concurrently resume work after I / O completes and begin to struggle for processor time, resulting in increased context switches per unit time and reduced scalability.
The Windows I / O subsystem (including device drivers) internally operates in asynchronous mode — a program can continue to execute concurrently with an I / O operation. Almost all modern hardware devices are of an asynchronous nature and do not need to be constantly polled to transfer data or determine when an I / O operation is complete.
Most devices support the ability Direct Memory Access (DMA) to transfer data between the device and the computer's RAM without requiring the participation of the processor in the operation, and generate an interrupt upon completion of the data transfer. Synchronous I / O, which is internally asynchronous, is only supported at the Windows application layer.
In Win32, asynchronous I / O is called overlapped I / O, a comparison of synchronous and overlapping I / O modes is shown in the figure below:
When an application makes an asynchronous request for an I / O operation, Windows either performs the operation immediately or returns a status code indicating that the operation is pending. After that, the thread of execution can start other I / O operations or perform some calculations. The programmer has several ways to organize the reception of notifications about the completion of I / O operations:
Win32 Event: An operation awaiting this event will be performed upon completion of I / O.
Calling a custom function using a mechanism Asynchronous Procedure Call (APC): The thread must be in an alertable wait state.
Receiving notifications through I / O Completion Ports (IOCP): this is usually the most efficient mechanism. We will explore it in detail below.
Certain I / O devices (for example, a file opened in unbuffered mode) can provide additional benefits if the application is able to maintain a small number of pending I / O requests at all times. To do this, it is recommended that you first make several requests for I / O operations and make a new request for each executed request. This will ensure that the next operation is initiated by the device driver as soon as possible, without waiting for the application to complete the next request. But do not overdo it with the amount of data transferred, as this will consume limited kernel memory resources.
I / O completion ports
Windows supports an efficient asynchronous I / O completion notification mechanism called I / O Completion Ports (IOCP). In .NET applications, it is available through the method ThreadPool.BindHandle ()... This mechanism is used by internal implementations of some types in .NET that perform I / O operations: FileStream, Socket, SerialPort, HttpListener, PipeStream, and some .NET Remoting pipes.
The IOCP mechanism shown in the figure above binds to multiple I / O descriptors (sockets, files, and specialized device driver objects) open asynchronously and to a specific thread of execution. As soon as the I / O operation associated with such a handle completes, Windows will add an alert to the appropriate IOCP port and pass the associated thread of execution for processing.
Using a pool of notifications and resuming threads that have initiated asynchronous I / O reduces the number of context switches per unit of time and increases processor utilization. Unsurprisingly, high-performance servers like Microsoft SQL Server use I / O completion ports.
Completion port is created by calling Win32 API function CreateIoCompletionPort that is passed the maximum concurrency value (number of threads), the completion key, and an optional handle to the I / O object. The termination key is a user-defined value that identifies various I / O descriptors. You can bind multiple descriptors to the same IOCP port by calling CreateIoCompletionPort again and passing in a handle to the existing completion port.
To establish communication with the specified IOCP port, user threads call the function GetCompletionStatus and await its completion. At any given time, a thread of execution can only be associated with one IOCP port.
Function call GetQueuedCompletionStatus blocks execution of the thread until it is notified (or the time-out), and then returns information about the completed I / O operation, such as the number of bytes transferred, the completion key, and the structure of the asynchronous I / O operation. If at the time of the notification all the threads associated with the I / O port are busy (that is, there are no threads waiting in the GetQueuedCompletionStatus call), the IOCP engine will create a new thread of execution, up to the maximum concurrency value. If the thread has called GetQueuedCompletionStatus and the notification queue is not empty, the function will return immediately without blocking the thread in the operating system kernel.
The IOCP engine is able to determine that some of the "busy" threads are actually performing synchronous I / O, and start an additional thread, possibly exceeding the maximum parallelism value. Notifications can also be sent manually, without performing I / O, by calling the function PostQueuedCompletionStatus.
The following code demonstrates an example of using ThreadPool.BindHandle () with a Win32 file descriptor:
Using System; using System.Threading; using Microsoft.Win32.SafeHandles; using System.Runtime.InteropServices; public class Extensions (internal static extern SafeFileHandle CreateFile (string lpFileName, EFileAccess dwDesiredAccess, EFileShare dwShareMode, IntPtr lpSecurityAttributes, ECreationDisposition dwCreationDisposition, EFileAttributes dwFlagsAndAttributes, IntPtr hTemplateFile); static unsafe extern bool WriteFile (SafeFileHandle hFile, byte lpBuffer, uint nNumberOfBytesToWrite, out uint lpNumberOfBytesWritten , System.Threading.NativeOverlapped * lpOverlapped); enum EFileShare: uint (None = 0x00000000, Read = 0x00000001, Write = 0x00000002, Delete = 0x00000004) enum ECreationDisposition: uint (New = 1, CreateAlways = 2, OpenExisting = 3, OpenAlways 4, TruncateExisting = 5) enum EFileAttributes: uint (// ... Some flags are not shown Normal = 0x00000080, Overlapped = 0x40000000, NoBuffering = 0x20000000,) enum EFileAccess: uint (// ... Some flags are not shown GenericRead = 0x80000000 , GenericWrite = 0x40000000,) static lo ng _numBytesWritten; // Brake for the write stream static AutoResetEvent _waterMarkFullEvent; static int _pendingIosCount; const int MaxPendingIos = 10; // Completion routine, called by I / O streams static unsafe void WriteComplete (uint errorCode, uint numBytes, NativeOverlapped * pOVERLAP) (_numBytesWritten + = numBytes; Overlapped ovl = Overlapped.Unpack (pOVERLAP); Notify the write stream that the number of pending I / O operations has decreased // to the allowed limit if (Interlocked.Decrement (ref _pendingIosCount) = MaxPendingIos) (_waterMarkFullEvent.WaitOne ();)))))
Let's look at the TestIOCP method first. This is where the CreateFile () function is called, which is a P / Invoke function used to open or create a file or device. To perform I / O operations asynchronously, you must pass the EFileAttributes.Overlapped flag to the function. If successful, the CreateFile () function returns a Win32 file descriptor that we bind to the I / O completion port by calling ThreadPool.BindHandle (). Next, an event object is created that is used to temporarily block the thread that initiated the I / O operation if there are too many such operations (the limit is set by the MaxPendingIos constant).
Then a cycle of asynchronous write operations begins. Each iteration creates a buffer with data to write and Overlapped structure containing an offset within the file (in this example, writing is always done at offset 0), an event descriptor passed on completion of the operation (not used by the IOCP mechanism), and an optional custom object IAsyncResult which can be used to pass state to a completion function.
Next, the Overlapped.Pack () method is called, which accepts a completion function and a data buffer. It creates an equivalent low-level structure for the I / O operation by placing it in unmanaged memory and pinning the data buffer. Freeing unmanaged low-level memory and unpinning the buffer must be done manually.
If there aren't too many I / O operations going on at the same time, we call WriteFile () passing it the specified low-level structure. Otherwise, we wait until an event appears indicating that the number of pending operations is less than the upper limit.
The WriteComplete completion function is called by a thread from the I / O completion thread pool as soon as the operation is complete. It is passed a pointer to a low-level asynchronous I / O structure that can be unpacked and converted into a managed Overlapped structure.
In summary, when dealing with high performance I / O devices, use asynchronous I / O operations with completion ports, either directly by creating and using a custom completion port in an unmanaged library, or by linking Win32 handles to a completion port in .NET using ThreadPool.BindHandle () method.
Thread pool in .NET
A thread pool in .NET can be used successfully for a variety of purposes, and different types of threads are created to achieve each of them. While discussing parallel computing, we previously got acquainted with the thread pool API, where we used it to parallelize computational tasks. However, thread pools can also be used to solve problems of a different kind:
Worker threads can handle asynchronous calls to custom delegates (such as BeginInvoke or ThreadPool.QueueUserWorkItem).
I / O completion threads can service notifications from the IOCP global port.
Wait threads can wait for registered events to occur, allowing you to wait for several events at once in one thread (using WaitForMultipleObjects), up to the Windows upper limit (maximum wait objects = 64). The event listener is used to provide asynchronous I / O without using completion ports.
Timer threads waiting for multiple timers to expire.
Gate threads control the processor utilization by threads from the pool, and also change the number of threads (within the established limits) to achieve the highest performance.
It is possible to initiate I / O operations that appear to be asynchronous but are not. For example, calling the ThreadPool.QueueUserWorkItem delegate and then performing a synchronous I / O operation is not a truly asynchronous operation and is no better than performing the same operation on a normal thread of execution.
Memory copy
It is not uncommon for a physical I / O device to return a buffer of data, which is copied over and over again until the application finishes processing it. Copying like this can take up a significant proportion of the processing power of the processor and should be avoided to maximize throughput. Next, we will look at a few situations when it is customary to copy data, and get acquainted with techniques to avoid this.
Unmanaged memory
Working with a buffer located in unmanaged memory is much more difficult in .NET than with a managed byte array, so programmers, in search of the simplest way, often copy the buffer into managed memory.
If the functions or libraries you are using allow you to explicitly specify the buffer in memory or pass your callback function to allocate the buffer, allocate the managed buffer and pin it in memory so that it can be accessed by both a pointer and a managed reference. If the buffer is large enough (> 85,000 bytes), it will be created at Large Object Heap so try to reuse existing buffers. If reusing a buffer is complicated by the uncertainty of the object's lifetime, use memory pools.
In other cases, when functions or libraries themselves allocate memory (unmanaged) for buffers, you can access that memory directly by pointer (from unsafe code) or using wrapper classes such as UnmanagedMemoryStream and UnmanagedMemoryAccessor... However, if you need to pass a buffer to some code that only operates on byte arrays or string objects, copying may be inevitable.
Even if you cannot avoid copying memory and some or most of your data is filtered early, unnecessary copying can be avoided by verifying the need for the data before copying it.
Exporting part of a buffer
Programmers sometimes assume that byte arrays contain only the data they need, from start to finish, forcing the calling code to split the buffer (allocate memory for the new byte array and copy only the data it needs). This situation can often be seen in protocol stack implementations. In contrast, equivalent unmanaged code can take a simple pointer, not even knowing whether it points to the beginning of the actual buffer or the middle of it, and a buffer length parameter to determine where the end of the data being processed is.
To avoid unnecessary memory copying, arrange for offset and length wherever you take a byte parameter. Use the length parameter instead of the Length property of the array, and add the offset value to the current indices.
Random read and merge write
Scatter Read and Merge Write is a Windows feature that reads or writes non-contiguous regions as if they were occupying a contiguous chunk of memory. This functionality in the Win32 API is provided in the form of functions ReadFileScatter and WriteFileGather... The Windows Sockets Library also supports the ability to read and write and write merge, providing its own functions: WSASend, WSARecv, and others.
Random read and merge write can be useful in the following situations:
When each packet has a fixed size header preceding the actual data. Random read and merge write can help you avoid having to copy headers whenever you want to get a contiguous buffer.
When you want to get rid of unnecessary system call overhead when doing I / O with multiple buffers.
Compared to the ReadFileScatter and WriteFileGather functions, which require each buffer to be exactly the size of one page, and the handle to be opened in asynchronous and unbuffered mode (which is an even bigger limitation), the socket-based read and write merge and scatter functions seem to be more practical. because they do not have these restrictions. .NET Framework Supports Bulb Read and Merge Writing for Sockets through Overloaded Methods Socket.Send () and Socket.Receive () without exporting generic read / write functions.
An example of using the read scatter and merge write functions can be found in the HttpWebRequest class. It concatenates the HTTP headers with the actual data without having to create a continuous buffer to store it.
File I / O
Typically, file I / O operations are performed through the file system cache, which provides some performance benefits: caching recently used data, read-ahead (read data from disk beforehand), lazy write (asynchronous write to disk), and concatenation of write operations for small chunks of data. ... By prompting Windows for the expected file access pattern, you can get additional performance gains. If your application is doing asynchronous I / O and is capable of addressing some of the buffering issues, then skipping the caching mechanism altogether may be a more efficient solution.
Caching management
When creating or opening files, programmers pass flags and attributes to the CreateFile function, some of which affect the behavior of the caching mechanism:
Flag FILE_FLAG_SEQUENTIAL_SCAN indicates that the file will be accessed sequentially, possibly skipping some parts, and random access is unlikely. As a result, the cache manager will look ahead and look beyond normal.
Flag FILE_FLAG_RANDOM_ACCESS indicates that the file will be accessed in no particular order. In this case, the cache manager will perform slightly ahead of the read, because of the reduced likelihood that the data read ahead will actually be required by the application.
Flag FILE_ATTRIBUTE_TEMPORARY indicates that the file is temporary, so the actual write operations to the physical medium (to prevent data loss) can be deferred.
In NET, these options are supported (except for the last one) by using the FileStream overloaded constructor that takes a parameter of the FileOptions enumeration type.
Random access has a negative impact on performance, especially when working with disk devices, as it requires moving heads. As technology advances, disk throughput has increased only by increasing storage density, not by decreasing latency. Modern disks are capable of reordering random access queries to reduce the overall time spent moving heads. This technique is called hardware command queuing (Native Command Queuing, NCO)... For this technique to be more effective, the disk controller needs to send several I / O requests at once. In other words, if possible, try to have multiple pending asynchronous I / O requests at once.
Unbuffered I / O
Unbuffered I / O operations are always performed without involving the cache. This approach has advantages and disadvantages. As with the cache control technique, unbuffered I / O is enabled through the "flags and attributes" parameter during file creation, but .NET does not provide access to this feature.
Flag FILE_FLAG_NO_BUFFERING disables caching for read and write operations, but does not affect the caching performed by the disk controller. This avoids copying (from the user buffer to the cache) and "pollution" of the cache (filling the cache with unnecessary data and displacing the necessary ones). However, unbuffered reads and writes must adhere to alignment requirements.
The following parameters must be equal to or multiples of the disk sector size: single transfer size, file offset, and memory buffer address. Typically, a disk sector is 512 bytes in size. The latest high-capacity disk drives have a sector size of 4096 bytes, but they can operate in compatibility mode by emulating 512-byte sectors (at the expense of performance).
Flag FILE_FLAG_WRITE_THROUGH instructs the cache manager to immediately flush writeable data from the cache (unless the FILE_FLAG_NO_BUFFERING flag is set) and tells the disk controller to write to the physical media immediately without storing the data in the intermediate hardware cache.
Read-ahead improves performance by utilizing more disk space even when the application reads synchronously with delays between operations. The correct definition of which part of the file an application asks for next is Windows-dependent. By disabling buffering, you also disable read-ahead, and must keep the disk device busy by performing multiple overlapping I / O operations.
Latency writes also improve the performance of applications that perform synchronous writes by giving the illusion that disk writes are very fast. The app will be able to improve CPU usage by blocking for shorter intervals. With buffering disabled, the write operations will take the full amount of time it takes to finish writing data to disk. Therefore, using asynchronous I / O with buffering disabled becomes even more important.
Asynchronous I / O using multiple threads
Overlapping and extended I / O allows for asynchronous I / O within a single thread, although the OS creates its own threads to support this functionality. Methods of this type are often used in one form or another in many early operating systems to support limited forms of asynchronous operations on single-threaded systems.
However, Windows provides multithreading support, so it becomes possible to achieve the same effect by performing synchronous I / O operations on multiple, independently executing threads. We've previously demonstrated these capabilities with multithreaded servers and grepMT (Chapter 7). In addition, threads provide a conceptually sequential and, presumably, much easier way to perform asynchronous I / O operations. As an alternative to the methods used in Programs 14.1 and 14.2, one could provide each thread with its own file descriptor, and then each of the threads could process every fourth record synchronously.
This way of using streams is demonstrated in the atouMT program, which is not included in the book but is included in the material posted on the Web site. Not only can atouMT run on any version of Windows, but it is simpler than either of the two asynchronous I / O programs because it is less complex to account for resource usage. Each thread simply maintains its own buffers on its own stack and performs a sequence of synchronous reads, conversions, and writes in a loop. At the same time, the program performance remains at a fairly high level.
Note
The atouMT.c program on the Web site comments on several possible pitfalls that may lie in wait for multiple threads accessing the same file at the same time. In particular, all individual file descriptors must be created using the CreateHandle function, not the DuplicateHandle function.
Personally, I prefer to use multi-threaded file processing over asynchronous I / O operations. Streams are easier to program and provide better performance in most cases.
There are two exceptions to this general rule. The first, as shown earlier in this chapter, deals with situations in which there can be only one outstanding operation, and a file descriptor can be used for synchronization purposes. A second and more important exception occurs in the case of asynchronous I / O completion ports, which will be discussed at the end of this chapter.
From the book Let's Build a Compiler! by Crenshaw Jack From the book Programming in Prolog author Kloxin W. From the book The C # 2005 Programming Language and the .NET 2.0 Platform. author Troelsen Andrew From the book The Informix Database Administrator's Guide. author Kustov Victor From the Microsoft Visual C ++ and MFC book. Programming for Windows 95 and Windows NT the author Frolov Alexander Vyacheslavovich2.2.3.2 Asynchronous I / O The server uses its own Asynchronous I / O (AIO) package, or the Kernel Asynchronous I / O (KAIO) package, if available, to speed up I / O. Custom I / O requests are processed asynchronously,
From the book Fundamentals of Object Oriented Programming by Meyer BertrandI / O As you know, operators<< и >> Shift the numeric value left and right by a specified number of bits. In the programs in this book, these operators are also used to enter information from the keyboard and display it on the screen.
From the book System Programming on Windows author Hart Johnson MInput and Output The two classes in the KERNEL library provide the basic input and output facilities: FILE and STD_FILES. The operations defined for a FILE object f include the following: create f.make ("name") - Links f to a file named name. f.open_write - Open f for writing f.open_read - Open f for
From the book Programming in Ruby [Ideology of the Language, Theory and Practice of Application] by Fulton HalCHAPTER 14 Asynchronous I / O and Completion Ports I / O operations are inherently slower than other types of processing. This slowdown is due to the following factors: Delays due to search time
From the book Programming in Prolog for Artificial Intelligence author Bratko Ivan10.1.7. Simple I / O You are already familiar with some of the I / O methods from the Kernel module; we called them without specifying the caller. These include the functions gets and puts, as well as print, printf, and p (the latter calls the inspect object's method to print it out in a
From the book The C Programming Language for the Personal Computer author Bochkov S.O. From the book Linux Programming By Example the author Robbins ArnoldChapter 6 Input and Output In this chapter, we will look at some of the built-in facilities for writing data to and reading from a file. Such tools can also be used to format the data objects of the program to obtain the desired form of their external representation.
From the book Java Programming Basics the author Sukhov S.A.Input and Output The input and output functions in the C Standard Library allow you to read data from or from input devices (such as a keyboard) and write data to files or output to various devices (such as a printer). withdrawal
From the book QT 4: GUI Programming in C ++ by Blanchette Jasmine4.4. Input and Output All Linux I / O operations are performed through file descriptors. This section introduces file descriptors, describes how to get and free them, and explains how to perform with them.
From the book The Ideal Programmer. How to become a software development professional the author Martin Robert S. From the author's bookChapter 12. I / O Almost every application has to read or write files or perform other I / O operations. Qt provides excellent I / O support with QIODevice, a powerful abstraction of "devices" that can read and write
From the author's bookInput and Output I also find it very important that my results are fed by appropriate input. Writing code is a creative job. Usually, my creativity is greatest when I encounter creative
I / O control.
block-oriented devices and byte oriented
Main idea
Key principle is device independence
Interrupt handling,
Device drivers,
It seems obvious that a wide variety of interrupts are possible for a variety of reasons. Therefore, a number is associated with an interrupt - the so-called interrupt number.
This number unambiguously corresponds to a particular event. The system is able to recognize interrupts and, when they occur, starts the procedure corresponding to the interrupt number.
Some interrupts (the first five in numerical order) are reserved for use by the central processor in case of any special events such as an attempt to divide by zero, overflow, etc. (these are really internal J interrupts).
Hardware interrupts always occur asynchronously to running programs. In addition, several interrupts can occur at the same time!
In order for the system not to "get lost" when deciding which interrupt to serve in the first place, there is a special priority scheme. Each interrupt is assigned a different priority. If several interrupts occur simultaneously, the system gives preference to the highest priority, postponing the processing of the remaining interrupts for a while.
The priority system is implemented on two Intel 8259 chips (or similar). Each microcircuit is an interrupt controller and serves up to eight priorities. Chips can be combined (cascaded) to increase the number of priority levels in the system.
Priority levels are abbreviated as IRQ0 - IRQ15.
24. Control of input / output. Synchronous and asynchronous I / O.
One of the main functions of the OS is to control all the input-output devices of the computer. The OS must transmit commands to devices, intercept interrupts and handle errors; it must also provide an interface between devices and the rest of the system. For development purposes, the interface should be the same for all types of devices (device independence). Learn more about I / O control in question 23.
Protection principles
Since the UNIX OS from its inception was conceived as a multiuser operating system, the problem of authorizing access of various users to files in the file system has always been relevant. By access authorization, we mean the actions of the system that allow or do not allow a given user to access this file, depending on the user's access rights and access restrictions set for the file. The access authorization scheme used in UNIX OS is so simple and convenient and at the same time so powerful that it has become the de facto standard of modern operating systems (which do not pretend to be systems with multilevel security).
File protection
As is customary in a multiuser operating system, UNIX maintains a uniform mechanism for controlling access to files and file system directories. Any process can access a file if and only if the access rights described in the file correspond to the capabilities of this process.
Protecting files from unauthorized access in UNIX is based on three facts. First, any process that creates a file (or directory) is associated with some system-unique user identifier (UID - User identifier), which can be further interpreted as the identifier of the owner of the newly created file. Second, there is a pair of identifiers associated with each process that tries to gain some access to the file - the current user and group identifiers. Third, each file is uniquely associated with its descriptor, the i-node.
The last fact is worth dwelling on in more detail. It is important to understand that filenames and files as such are not the same thing. In particular, when there are multiple hard links to the same file, the multiple file names actually represent the same file and are associated with the same i-node. Any i-node used in the file system is always uniquely associated with one and only one file. The I-node contains a lot of various information (most of it is available to users through the stat and fstat system calls), and among this information there is a part that allows the file system to evaluate the authority of a given process to access this file in the required mode.
The general principles of protection are the same for all existing variants of the system: The information of the i-node includes the UID and GID of the current owner of the file (immediately after the file is created, the identifiers of its current owner are set by the corresponding valid identifier of the creator process, but can later be changed by the chown and chgrp system calls) ... In addition, the i-node of the file stores a scale that indicates what the user - its owner can do with the file, what users from the same user group as the owner can do with the file, and what others can do with the file. users. The small details of the implementation differ in different versions of the system.
28. Controlling access to files in Windows NT. Access rights lists.
The access control system in Windows NT is highly flexible, which is achieved through a wide variety of subjects and access objects, as well as granularity of access operations.
File access control
For shared resources, Windows NT uses a common object model that contains security characteristics such as a set of allowed operations, an owner ID, and an ACL.
Objects in Windows NT are created for any resources when they are or become shared - files, directories, devices, memory sections, processes. Characteristics of objects in Windows NT are divided into two parts - the general part, the composition of which does not depend on the type of the object, and the individual part, determined by the type of the object.
All objects are stored in tree-like hierarchical structures, the elements of which are branch objects (directories) and leaf objects (files). For file system objects, this relationship is a direct reflection of the directory and file hierarchy. For objects of other types, the hierarchical diagram of relations has its own content, for example, for processes it reflects parent-child relationships, and for devices it reflects belonging to a certain type of device and the connection of a device with other devices, for example, a SCSI controller with disks.
Checking access rights for objects of any type is performed centrally using the Security Reference Monitor, which operates in privileged mode.
The Windows NT security system is characterized by a large number of different predefined (built-in) access principals - both individual users and groups. So, the system always has users such as Adininistrator, System and Guest, as well as the Users, Adiniiiistrators, Account Operators, Server Operators, Everyone and others groups. The meaning of these built-in users and groups is that they are endowed with some rights, making it easier for the administrator to create an effective access control system. When adding a new user, the administrator only has to decide which group or groups to assign this user to. Of course, the administrator can create new groups, as well as add rights to the built-in groups to implement their own security policy, but in many cases the built-in groups are enough.
Windows NT supports three classes of access operations, which differ in the type of subjects and objects involved in these operations.
□ Permissions are a set of operations that can be defined for subjects of all types in relation to objects of any type: files, directories, printers, memory sections, etc. Permissions in their purpose correspond to the access rights to files and directories in QC UNIX.
□ User rights - defined for subjects of the group type to perform some system operations: setting the system time, archiving files, shutting down the computer, etc. These operations involve a special access object - the operating system as a whole.
It is mainly rights, not permissions, that distinguish one built-in user group from another. Some rights for a built-in group are also built-in - they cannot be removed from this group. The rest of the built-in group rights can be removed (or added from the general list of rights).
□ User abilities are determined for individual users to perform actions related to the formation of their operating environment, for example, changing the composition of the main menu of programs, the ability to use the Run menu item, etc. available to the user by default), the administrator can force the user to work with the operating environment that the administrator considers most appropriate and protects the user from possible errors.
The rights and permissions given to the group are automatically granted to its members, allowing the administrator to treat a large number of users as a unit of accounting information and minimize their actions.
When a user logs on to the system, a so-called access token is created for him, which includes the user ID and identifiers of all the groups the user belongs to. The token also contains: a default access control list (ACL), which consists of permissions and is applied to objects created by the process; a list of user rights to perform system actions.
All objects, including files, streams, events, even access tokens, when they are created, are supplied with a security descriptor. The security descriptor contains an access control list - ACL.
File descriptor is a non-negative integer assigned by the OS to the file opened by the process.
ACL(eng. Access Control List- an access control list, pronounced "ecl" in English) - determines who or what can get access to a particular object, and what operations are allowed or prohibited by this subject to perform over the object.
ACLs are the backbone of selective access control systems. ( Wiki)
The owner of an object, usually the user who created it, has the right to selectively control access to the object and can change the object's ACL to allow or prevent others from accessing the object. The built-in Windows NT administrator, unlike the UNIX superuser, may not have some permissions on the object. To implement this feature, administrator and administrator group IDs can be included in the ACL, as well as ordinary user IDs. However, the administrator still has the ability to perform any operations with any objects, since he can always take ownership of the object, and then, as the owner, receive the full set of permissions. However, the administrator cannot return ownership to the previous owner of the object, so the user can always find out that the administrator was working with his file or printer.
When a process requests an object access operation in Windows NT, control is always passed to the Security Monitor, which compares the user and user group IDs from the access token with the IDs stored in the object's ACL elements. Unlike UNIX, Windows NT ACLs can contain both allowed lists and lists of prohibited operations for the user.
Windows NT uniquely defines the rules by which an ACL is assigned to a newly created object. If the caller explicitly sets all access rights to the newly created object during object creation, the security system assigns this ACL to the object.
If the caller does not supply the object with an ACL and the object has a name, then the principle of inherited permissions applies. The security system looks at the ACL of the object directory that stores the name of the new object. Some of the ACL entries in the object directory can be marked as inherited. This means that they can be assigned to new objects created in this directory.
When the process has not explicitly set an ACL for the object being created, and the directory object has no inherited ACLs, the default ACL from the process's access token is used.
29. The Java programming language. Java virtual machine. Java technology.
Java is an object-oriented programming language developed by Sun Microsystems. Java applications are usually compiled to special bytecode, so they can run on any Java virtual machine (JVM) regardless of the computer architecture. Java programs are translated into bytecode executed by the Java Virtual Machine ( JVM) - a program that processes the byte code and sends instructions to the equipment as an interpreter, but with the difference that the byte code, unlike text, is processed much faster.
The advantage of this way of executing programs is that the bytecode is completely independent of the operating system and hardware, which allows you to run Java applications on any device for which there is a corresponding virtual machine. Another important feature of Java technology is flexible security system due to the fact that the execution of the program is completely controlled by the virtual machine... Any operations that exceed the set permissions of the program (for example, an attempt to unauthorized access to data or to connect to another computer) will cause an immediate interruption.
Often the disadvantages of the concept of a virtual machine include the fact that the execution of bytecode by a virtual machine can reduce the performance of programs and algorithms implemented in the Java language.
Java Virtual Machine(short for Java VM, JVM) - Java virtual machine - the main part of the Java runtime system, the so-called Java Runtime Environment (JRE). The Java Virtual Machine interprets and executes Java bytecode that is previously generated from the Java source code by the Java compiler (javac). The JVM can also be used to run programs written in other programming languages. For example, Ada source code can be compiled into Java bytecode, which can then be executed by the JVM.
The JVM is a key component of the Java platform. Since Java virtual machines are available for many hardware and software platforms, Java can be viewed both as middleware and as a standalone platform, hence the write once, run anywhere principle. Using a single bytecode for multiple platforms allows Java to be described as “compile once, run anywhere”.
Runtime environment
Programs intended to run on the JVM must be compiled in a standardized portable binary format, usually represented as .class files. A program can consist of many classes located in different files. To facilitate the placement of large programs, some of the .class files can be packaged together in a so-called .jar file (short for Java Archive).
The JVM virtual machine executes .class or .jar files by emulating instructions written for the JVM by interpreting or using a just-in-time (JIT) compiler such as Sun microsystems' HotSpot. JIT compilation is used by most JVMs these days for the sake of speed.
Like most virtual machines, the Java Virtual Machine has a stack-oriented architecture common to microcontrollers and microprocessors.
The JVM, which is an instance of the JRE (Java Runtime Environment), comes into play when Java programs are executed. Upon completion of execution, this instance is discarded by the garbage collector. JIT is part of the Java virtual machine that is used to speed up the execution time of applications. JIT simultaneously compiles portions of bytecode that have similar functionality, and therefore reduces the amount of compilation time.
j2se (java 2 standard edition) - the standard library includes:
GUI, NET, Database ...
30. The .NET platform. Basic ideas and provisions. .NET programming languages.
.NET Framework- software technology from Microsoft, designed to create common programs and web applications.
One of the main ideas of Microsoft .NET is the interoperability of different services written in different languages. For example, a service written in C ++ for Microsoft .NET can access a class method from a library written in Delphi; in C #, you can write a class that inherits from a class written in Visual Basic .NET, and an exception thrown by a method written in C # can be caught and handled in Delphi. Each library (assembly) in .NET has information about its version, which allows you to eliminate possible conflicts between different versions of assemblies.
Applications can also be developed in a text editor and use the console compiler.
Similar to Java technology, the .NET development environment generates bytecode for execution by a virtual machine. The input language of this machine in .NET is called MSIL (Microsoft Intermediate Language), or CIL (Common Intermediate Language, later), or simply IL.
The use of bytecode allows you to get cross-platform at the level of the compiled project (in terms of .NET: assembly), and not only at the level of the source code, as, for example, in C. Before starting the assembly in the CLR runtime, the bytecode is converted by the built-in JIT compiler (just in time, compilation on the fly) into machine codes of the target processor. It is also possible to compile an assembly into native code for a selected platform using the NGen.exe utility supplied with the .NET Framework.
During the execution of the translation procedure, the source code of the program (written in SML, C #, Visual Basic, C ++, or any other programming language that is supported by .NET) is converted by the compiler into a so-called assembly and saved as a dynamically linked library (Dynamically Linked Library, DLL) or executable file (Executable, EXE).
Naturally, for each compiler (whether it is a C # compiler, csc.exe or Visual Basic, vbc.exe), the run-time environment performs the necessary mapping of the used types into CTS types, and the program code - into the code of the .NET "abstract machine" - MSIL (Microsoft Intermediate Language).
As a result, a software project is formed in the form of an assembly - a self-contained component for deployment, replication and reuse. An assembly is identified by the digital signature of the author and a unique version number.
Built-in programming languages (shipped with the .NET Framework):
C #; J #; VB.NET; JScript .NET; C ++ / CLI - new version of C ++ (Managed).
31. Functional components of the OS. File management
OS functional components:
The functions of the operating system of a stand-alone computer are usually grouped either according to the types of local resources that the OS manages or according to specific tasks that apply to all resources. These function groups are sometimes referred to as subsystems. The most important subsystems for resource management are the subsystems for managing processes, memory, files, and external devices, and the subsystems common to all resources are the subsystems of the user interface, data protection, and administration.
File management:
The ability of the OS to "shield" the complexities of real hardware is very clearly manifested in one of the main subsystems of the OS - the file system.
The file system links the storage medium on one side and the API (Application Programming Interface) for accessing the files on the other. When an application program accesses a file, it has no idea how the information is located in a particular file, as well as on what physical type of media (CD, hard disk, magnetic tape or flash memory block) it is recorded. All the program knows is the name of the file, its size and attributes. It receives this data from the file system driver. It is the file system that determines where and how the file will be written to the physical medium (for example, a hard disk).
From the point of view of the operating system, the entire disk is a set of clusters of 512 bytes and above. File system drivers organize clusters into files and directories (which are actually files containing a list of files in that directory). These same drivers keep track of which of the clusters are currently in use, which are free, which are marked as faulty.
However, the file system is not necessarily directly related to the physical storage medium. There are virtual file systems, as well as network file systems, which are just a way to access files on a remote computer.
In the simplest case, all files on a given disk are stored in one directory. This sibling scheme was used in CP / M and in the first version of MS-DOS 1.0. A hierarchical file system with directories nested inside each other first appeared in Multics, then in UNIX.
Directories on different disks can form several separate trees, as in DOS / Windows, or they can be combined into one tree common to all disks, as in UNIX-like systems.
In fact, in DOS / Windows systems, as well as in UNIX-like systems, there is one root directory with nested directories named "c:", "d:", etc. Hard disk partitions are mounted in these directories. That is, c: \ is just a link to file: /// c: /. However, unlike UNIX-like file systems, writing to the root directory is prohibited in Windows, as well as viewing its contents.
On UNIX, there is only one root directory, and all other files and directories are nested within it. To access files and directories on any disk, you need to mount this disk with the mount command. For example, to open files on a CD, you just need to tell the operating system, “take the file system on this CD and show it in the / mnt / cdrom directory”. All files and directories on the CD will appear in this / mnt / cdrom directory called the mount point. On most UNIX-like systems, removable disks (floppy disks and CDs), flash drives, and other external storage devices are mounted in the / mnt, / mount, or / media directory. Unix and UNIX-like operating systems also allow disks to be automatically mounted when the operating system boots.
Note the use of slashes in Windows, UNIX and UNIX-like operating systems (Windows uses a backslash "\", and UNIX and UNIX-like operating systems use a simple slash "/")
In addition, it should be noted that the above system allows you to mount not only the file systems of physical devices, but also individual directories (option --bind) or, for example, an ISO image (option loop). Add-ons such as FUSE also allow you to mount, for example, an entire directory on FTP and a very large number of different resources.
An even more complex structure is used in NTFS and HFS. In these file systems, each file is a set of attributes. Attributes are considered not only traditional read-only, system, but also the file name, size and even content. Thus, for NTFS and HFS, what is stored in the file is just one of its attributes.
If you follow this logic, a single file can contain multiple content variations. Thus, multiple versions of the same document can be stored in one file, as well as additional data (the file icon, the program associated with the file). This arrangement is typical of HFS on the Macintosh.
32. Functional components of the OS. Process management.
Process management:
The most important part of the operating system, which directly affects the functioning of a computer, is the process control subsystem. A process (or in other words, a task) is an abstraction that describes a running program. For the operating system, a process is a unit of work, an application for the consumption of system resources.
In a multitasking (multiprocessing) system, a process can be in one of three main states:
RUNNING - active state of the process, during which the process has all the necessary resources and is directly executed by the processor;
WAITING - a passive state of the process, the process is blocked, it cannot be executed for its internal reasons, it waits for some event to take place, for example, completion of an I / O operation, receiving a message from another process, freeing some resource it needs;
READY is also a passive state of the process, but in this case the process is blocked due to circumstances external to it: the process has all the resources it needs, it is ready to run, but the processor is busy executing another process.
During the life cycle, each process moves from one state to another in accordance with the process scheduling algorithm implemented in this operating system.
CP / M standard
The creation of operating systems for microcomputers was initiated by the OS SR / M. It was developed in 1974 and has since been installed on many 8-bit machines. Within the framework of this operating system, a significant amount of software was created, including translators from the languages BASIC, Pascal, C, Fortran, Cobol, Lisp, Ada and many others, text. They allow you to prepare documents much faster and more conveniently than using a typewriter.
MSX standard
This standard defined not only the OS, but also the characteristics of the hardware for school PCs. According to the MSX standard, the machine had to have at least 16 K of RAM, 32 K read-only memory with a built-in BASIC interpreter, a color graphic display with a resolution of 256x192 pixels and 16 colors, a three-channel sound generator for 8 octaves, a parallel port for connecting a printer. and a controller for controlling an external storage device connected externally.
The operating system of such a machine should have the following properties: the required memory is no more than 16 K, compatibility with CP / M at the level of system calls, compatibility with DOS in terms of file formats on external drives based on floppy disks, support for translators of BASIC, C languages, Fortran and Lisp.
Pi - system
In the initial period of development of personal computers, the USCD p-system operating system was created. The basis of this system was the so-called P-machine - a program that emulates a hypothetical universal computing machine. A P-machine simulates the operation of a processor, memory, and external devices by executing special commands called P-code. P-system software components (including compilers) are written in P-code, application programs are also compiled into P-code. Thus, the main distinguishing feature of the system was the minimum dependence on the features of the PC equipment. This is what ensured the portability of the Pi-system to various types of machines. The compactness of the P-code and the conveniently implemented paging mechanism made it possible to execute relatively large programs on a PC with a small RAM.
I / O control.
I / O devices are divided into two types: block-oriented devices and byte oriented devices. Block-oriented devices store information in fixed-size blocks, each with its own address. The most common block-oriented device is the disk. Byte-oriented devices are not addressable and do not allow a search operation; they generate or consume a sequence of bytes. Examples are terminals, line printers, network adapters. The electronic component is called a device controller or adapter. The operating system deals with the controller. The controller performs simple functions, monitors and corrects errors. Each controller has several registers that are used to communicate with the central processor. The OS performs I / O by writing commands to the controller registers. The floppy disk controller of the IBM PC accepts 15 commands such as READ, WRITE, SEEK, FORMAT, etc. When the command is accepted, the processor leaves the controller and does other work. At the end of the command, the controller organizes an interrupt in order to transfer control of the processor to the operating system, which must check the results of the operation. The processor obtains the results and device status by reading information from the controller registers.
Main idea I / O software organization consists in dividing it into several levels, and the lower levels provide shielding of the features of the equipment from the upper ones, and those provide a convenient interface for users.
Key principle is device independence... The type of program should not depend on whether it reads data from a floppy disk or from a hard disk. Another important issue for I / O software is error handling. In general, errors should be handled as close to the hardware as possible. If the controller detects a read error, then it should try to correct it. If it does not succeed, then the device driver should deal with the error correction. And only if the lower level cannot handle the error, it reports the error to the upper level.
Another key issue is the use of blocking (synchronous) and non-blocking (asynchronous) transfers. Most physical I / O operations are performed asynchronously — the processor starts transferring and moves on to other work until an interrupt occurs. It is necessary that the I / O operations are blocking - after the READ command, the program is automatically suspended until the data enters the program buffer.
The last problem is that some devices are shared (disks: simultaneous access of multiple users to the disk is not a problem), while others are dedicated (printers: you cannot mix lines printed by different users).
To solve the problems posed, it is advisable to divide the I / O software into four layers (Figure 2.30):
Interrupt handling,
Device drivers,
Device independent layer of the operating system,
· Custom software layer.
The concept of a hardware interrupt and its handling.
Asynchronous or external (hardware) interrupts are events that come from external sources (for example, peripheral devices) and can occur at any arbitrary moment: a signal from a timer, network card or disk drive, keyboard keystrokes, mouse movement; They require immediate reaction (processing).
Almost all I / O systems in a computer operate using interrupts. In particular, when you press keys or click the mouse, the hardware generates interrupts. In response to them, the system, respectively, reads the code of the pressed key or memorizes the coordinates of the mouse cursor. Interrupts are generated by the disk controller, LAN adapter, serial ports, audio adapter, and other devices.
We've been waiting for him too long
What could be more stupid than waiting?
B. Grebenshchikov
In this lecture, you will learn
Using the select system call
Using the poll system call
Some aspects of using select / poll in multithreaded programs
Standard asynchronous I / O facilities
The select system call
If your program is primarily concerned with I / O operations, you can get the most important of the benefits of multithreading in a single-threaded program by using the select (3C) system call. On most Unix systems select is a system call, or at any rate is described in section 2 (system calls) of the system manual, i.e. the link to it should look like select (2), but in Solaris10 the corresponding system manual page is located in section 3C (the standard C library).
I / O devices are usually much slower than the CPU, so the CPU usually has to wait for them to perform operations on them. Therefore, in all operating systems, synchronous I / O system calls are blocking operations.
This also applies to network communications - interaction via the Internet is associated with high delays and, as a rule, occurs through a not very wide and / or congested communication channel.
If your program works with several I / O devices and / or network connections, it is not profitable for it to block on an operation associated with one of these devices, because in this state it may miss the opportunity to perform I / O from another device without blocking. This problem can be solved by creating threads that work with different devices. In the previous lectures, we learned everything you need to develop such programs. However, there are other remedies to solve this problem.
The select (3C) system call allows you to wait for multiple devices or network connections to be ready (actually, on most object types that can be identified by a file descriptor). When one or more of the descriptors are ready to transfer data, select (3C) returns control to the program and passes the lists of ready descriptors in the output parameters.
Select (3C) uses sets (sets) of descriptors as parameters. On older Unix systems, sets were implemented as 1024-bit bit masks. In modern Unix systems and in other operating systems that implement select, sets are implemented as an opaque type fd_set, over which some set-theoretic operations are defined, namely, clearing a set, including a descriptor in a set, excluding a descriptor from a set, and checking for a descriptor in a set. The preprocessor directives for performing these operations are described on the select (3C) man page.
On 32-bit UnixSVR4, including Solaris, fd_set is still a 1024-bit mask; on 64-bit versions of SVR4, this is a 65536-bit bit mask. The size of the mask determines not only the maximum number of file descriptors in the set, but also the maximum number of the file descriptor in the set. The size of the mask on your version of the system can be determined at compile time by the value of the FD_SETSIZE preprocessor symbol. File descriptors are numbered in Unix at 0, so the maximum descriptor number is FD_SETSIZE-1.
Thus, if you are using select (3C), you need to set limits on the number of descriptors in your process. This can be done with the shell command ulimit (1) before starting the process, or with the setrlimit (2) system call while your process is running. Of course, setrlimit (2) must be called before you start creating file descriptors.
If you need to use more than 1,024 descriptors in a 32-bit program, Solaris10 provides a transitional API. To use it, you need to define
preprocessor symbol FD_SETSIZE with a numeric value greater than 1024 before including the file
In some implementations, fd_set is implemented differently, without using bit masks. For example, Win32 provides select as part of the so-called WinsockAPI. Win32fd_set is implemented as a dynamic array containing file descriptor values. Therefore, you should not rely on knowledge of the internal structure of the fd_set type.
However, changing the size of the fd_set bitmask or the internal representation of this type requires recompilation of all programs using select (3C). In the future, when the architectural limit of 65536 descriptors per process is increased, a new version of the fd_set and select implementation and a new recompilation of programs may be required. To avoid this and simplify the transition to the new version of ABI, SunMicrosystems recommends that you do not use select (3C) and use the poll (2) system call instead. The poll (2) system call is discussed later in this chapter.
The select (3C) system call has five parameters.
intnfds is a number one more than the maximum file descriptor number in all sets passed as parameters.
fd_set * readfds - Input parameter, set of descriptors that should be checked for readability. Ending a file or closing a socket is considered a special case of readiness. Regular files are always considered ready to be read. Also, if you want to test a TCP listening socket for accept (3SOCKET), it should be included in this set. Also, an output parameter, a set of descriptors ready to be read.
fd_set * writefds - Input parameter, set of descriptors that should be checked for readiness for writing. A delayed write error is considered a special case of write-ready. Regular files are always ready to be written. Also, if you want to check the completion of an asynchronousconnect (3SOCKET) operation, the socket should be included in this set. Also, an output parameter, a set of descriptors ready to be written.
fd_set * errorfds - Input parameter, set of descriptors to check for exceptional conditions. The definition of an exception depends on the type of file descriptor. For TCP sockets, an exception occurs when out-of-band data arrives. Regular files are always considered to be in an exceptional state. Also, an output parameter, a set of descriptors on which exceptions occurred.
structtimeval * timeout– timeout, time interval specified with microsecond precision. If this parameter is NULL, then select (3C) will wait indefinitely; if a zero time interval is specified in the structure, select (3C) operates in polling mode, that is, it returns control immediately, possibly with empty descriptor sets.
Instead of all parameters of type fd_set *, you can pass a null pointer. This means that we are not interested in the corresponding event class. Select (3C) returns the total number of ready handles in all sets on normal completion (including when it timed out), and -1 on error.
Example 1 shows how to use select (3C) to copy data from a network connection to a terminal, and from a terminal to a network connection. This program is simplified, it assumes that writing to the terminal and to the network connection will never be blocked. Since both the terminal and the network connection have internal buffers, this is usually the case with small data streams.
Example 1. Two-way copying of data between a terminal and a network connection. An example is taken from the book of W.R. Stevens, Unix: Network Application Development. Instead of standard system calls, "wrappers" are used, described in the file "unp.h"
#include "unp.h"
void str_cli (FILE * fp, int sockfd) (
int maxfdp1, stdineof;
char sendline, recvline;
if (stdineof == 0) FD_SET (fileno (fp), & rset);
FD_SET (sockfd, & rset);
maxfdp1 = max (fileno (fp), sockfd) + 1;
Select (maxfdp1, & rset, NULL, NULL, NULL);
if (FD_ISSET (sockfd, & rset)) (/ * socket is readable * /
if (Readline (sockfd, recvline, MAXLINE) == 0) (
if (stdineof == 1) return; / * normal termination * /
else err_quit ("str_cli: server terminated prematurely");
Fputs (recvline, stdout);
if (FD_ISSET (fileno (fp), & rset)) (/ * input is readable * /
if (Fgets (sendline, MAXLINE, fp) == NULL) (
Shutdown (sockfd, SHUT_WR); / * send FIN * /
FD_CLR (fileno (fp), & rset);
Writen (sockfd, sendline, strlen (sendline));
Note that the program in Example 1 recreates the descriptor sets before each call to select (3C). This is necessary because select (3C) will modify its parameters on normal completion.
select (3C) is considered MT-Safe, however, when using it in a multithreaded program, the following point should be borne in mind. Indeed, select (3C) itself does not use local data and therefore calling it from multiple threads should not lead to problems. However, if multiple threads are handling overlapping file descriptor sets, the following scenario is possible:
Thread 1 calls read from descriptor s and receives all data from its buffer.
Thread 2 calls read from handle s and blocks.
To avoid this scenario, working with file descriptors in such conditions should be protected by mutexes or some other mutually exclusive primitives. It is important to emphasize that it is not select that needs to be protected, but the sequence of operations on a specific file descriptor, starting with including the descriptor in the set for select and ending with receiving data from this descriptor, more precisely, updating pointers in the buffer into which you read this data. If this is not done, even more exciting scenarios are possible, for example:
Thread 1 includes the s descriptor in readfds and calls select.
select on thread 1 returns s as readable
Thread 2 includes the s descriptor in the readfds set and calls select
select on thread 2 returns s as readable
Thread 1 calls read from descriptor s and receives only a portion of the data from its buffer.
Thread 2 calls read from descriptor s, receives data and writes it over the data received by thread 1
In Chapter 10, we'll look at the architecture of an application in which multiple threads share a common pool of file descriptors — the so-called workerthreads architecture. In this case, the threads, of course, must indicate to each other which descriptors they are currently working with.
From a multithreaded program development perspective, an important disadvantage of select (3C) - or perhaps the disadvantage of POSIXThreadAPI - is the fact that POSIX synchronization primitives are not file descriptors and cannot be used in select (3C). At the same time, in the real development of multithreaded I / O programs, it would often be useful to wait in one operation for file descriptors and other threads to wait for their own process.