Proposed IOCTL callback extension

[jimuk]

I have just started to explore the Dokan project so please excuse me if this suggestion has been made before.

It would be greatl if the DOKAN_OPERATIONS structure was extended to provide an optional callback method for passing arbitrary IOCTL messages to the underlying file system code. For instance:

NTSTATUS(* DeviceIoControl )(LPCWSTR FileName, DWORD dwIoControlCode, LPVOID lpInBuffer, DWORD nInBufferSize, LPVOID lpOutBuffer, DWORD nOutBufferSize, LPDWORD lpBytesReturned, PDOKAN_FILE_INFO DokanFileInfo)

I would suggest this could have several uses/advantages:

a. Allow arbitrary data to be "tunneled" to the underlying file system code

b. Provide a way to implement application specific features without changing Dorkan.

c. Implement this in a Windows compatible way

Jim

[Liryna]

Hi @jimuk ,

Dokan is made to create filesystem. How IOCTL callback extension would help to do this ?

[jimuk]

Hi @Liryna,

Dorkan is great and has a very clear and straight forward API. This allows almost any information to be represented as a Windows friendly file system and explored/managed using familiar tools like Explorer.

However, the flip side of this simplicity is that there are a very limited number of ways to interact with a file and it's metadata. For instance, SetFileTime() supports exactly 3 timestamps but some file systems may have more (NTFS has 4 or more if you could additional "hidden" metadata).

The IOCTL interface would would allow Dorkan to be used as the basis for more complex integrations. At a basic level information for the underlying system (think SQL or another "alien" file system) would appear as a regular Windows file system but applications on top of this layer could use the IOCTL mechanism to query additional features and metadata by "tunneling" this information through Dorkan.

I think I could probably write a patch to add this feature but before I started on this I wanted to check if it t had been discussed before and if it was likely to be accepted by the community.

I hope this helps.

Jim

[Rondom]

I don't have any clue about Windows programming. What is the difference between IOCTLs and FSCTLs and why would one use one over the other?

[jimuk]

I don't think there is much practical difference between them. I understand that FSCTL codes are used to communicate with file system drivers whilst IOCTL are more general and used to communicate with any driver. Basically, they are a way for non-data IO with the underlying driver. The method call for the IOCTL allows a buffer to be passed down to the driver and a buffer received back. Both are optional and the communication can be one way if require or even pass no data at all. The Linux ioctl() works in a similar way.

Jim

[Rondom]

I think it sounds good. @Liryna, what do you think? I think it could be useful to support both IOCTLs and FSCTLs if it is not too much extra work.

[Liryna]

This can be a good idea if it is asked by the community 👍
@jimuk I do not know how you would like to implement it but I would be really great to no break current kernel <-> library communication or otherwise this will need to be moved to the 2.x.x version.

[jimuk]

If I understand the Dokan API it will require an extra entry in the DOKAN_OPERATIONS structure. Would this break the kernel <-> library communication?

BTW: I haven't read about future plans for Dokan but maybe the structure could be replaced by providing functions to "register" handlers with the driver code. This would make it more extensible in the future and provide some backwards compatibility with older drivers?

Jim

[marinkobabic]

The question is about how many additional IOCTL are we talking here which are actually not implement in Dokan? Depending on how big the list is it may make sense just to implement them. Don't think that a file system changes the IOCTL that much in future. Can you provide such a list please? Am 13.07.2017 23:04 schrieb "jimuk" <[email protected]>:

…

If I understand the Dokan API it will require an extra entry in the DOKAN_OPERATIONS structure. Would this break the kernel <-> library communication? BTW: I haven't read about future plans for Dokan but maybe the structure could be replaced by providing functions to "register" handlers with the driver code. This would make it more extensible in the future and provide some backwards compatibility with older drivers? Jim — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#544 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADHSOENsJazm-EbHx6tkVkPgJM6yDlbAks5sNoZigaJpZM4OKc5o> .

[magol]

I think this is probably a good idea. The file system that we implement is quite different from NTFS. There are many metadata that we can not convey through the file system, but that we really want to use. If it had been great to solve this in some way.

And I think it is a great idea to get rid of DOKAN_OPERATIONS to next version and replace it with any kind of registration as @jimuk suggests. As it is now, it is not possible to add any new functionality without destroy backward compatibility.

[jimuk]

To answer the points made by @marinkobabic and @Rondom. This change only requires one new callback handler adding to driver. This is a generic mechanism that any IOCTL/FSCTL (they are essentially the same) can be tunneled through. The IOCTL mechanism supports three modes of operation:

1. Read - Pull data from underlying driver
2. Write - Push data down to underlying driver
3. Read-Write - Simultaneous of both above

The following MSDN page explains how Windows uses this:

[(https://msdn.microsoft.com/en-us/library/windows/desktop/aa363219(v=vs.85).aspx)]

Dokan does not yet support async IO so I would suggest we drop the lpOverlapped member. This would leave the following handler signature:

NTSTATUS(* DeviceIoControl )(LPCWSTR FileName, DWORD dwIoControlCode, LPVOID lpInBuffer, DWORD nInBufferSize, LPVOID lpOutBuffer, DWORD nOutBufferSize, LPDWORD lpBytesReturned, PDOKAN_FILE_INFO DokanFileInfo)

Jim

[marinkobabic]

What happens when you use one of the already implemented IOCTL by Dokan? What IOCTL implementations are you missing actually? You request is in generally: please provide me a way in user Mode to talk directly to the operating system. Am 14.07.2017 10:39 schrieb "jimuk" <[email protected]>:

…

To answer the points made by @marinkobabic <https://github.com/marinkobabic> and @Rondom <https://github.com/rondom>. This change only requires one new callback handler adding to driver. This is a generic mechanism that any IOCTL/FSCTL (they are essentially the same) can be tunneled through. The IOCTL mechanism supports three modes of operation: 1. Read - Pull data from underlying driver 2. Write - Push data down to underlying driver 3. Read-Write - Simultaneous of both above The following MSDN page explains how Windows uses this: [(https://msdn.microsoft.com/en-us/library/windows/desktop/ aa363219(v=vs.85).aspx)] Dokan does not yet support async IO so I would suggest we drop the lpOverlapped member. This would leave the following handler signature: NTSTATUS(* DeviceIoControl )(LPCWSTR FileName, DWORD dwIoControlCode, LPVOID lpInBuffer, DWORD nInBufferSize, LPVOID lpOutBuffer, DWORD nOutBufferSize, LPDWORD lpBytesReturned, PDOKAN_FILE_INFO DokanFileInfo) Jim — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#544 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADHSOPUVLDAVFMyVlRsdOjbebq0SSZssks5sNylWgaJpZM4OKc5o> .

[jimuk]

1. Example user mode code to fetch an arbitrary 100 bytes from the underlying driver could be:

CHAR cBuffer[100];
DWORD dwBytesRead;

BOOL bResult=WINAPI DeviceIoControl(hDevice, MY_CONTROL_CODE, cBuffer, 100, NULL, 0, &dwBytesRead, NULL);

2. Dokan would tunnel this down to the kernel mode driver and then back to the user mode file system code. It would be passed to a method like:

NTSTATUS ProcessDeviceIoControl(LPCWSTR FileName, DWORD dwIoControlCode, LPVOID lpInBuffer, DWORD nInBufferSize, LPVOID lpOutBuffer, DWORD nOutBufferSize, LPDWORD lpBytesReturned, PDOKAN_FILE_INFO DokanFileInfo)

3. This method would resolve the dwIoControl code, perform the desired operation and return the requested bytes.

Jim

[marinkobabic]

This means you are here talking always about custom IOCTL codes and not about the official available codes? Am 14.07.2017 18:48 schrieb "jimuk" <[email protected]>:

…

1. Example user mode code to fetch an arbitrary 100 bytes from the underlying driver could be: `CHAR cBuffer[100]; DWORD dwBytesRead; BOOL bResult=WINAPI DeviceIoControl(hDevice, MY_CONTROL_CODE, cBuffer, 100, NULL, 0, &dwBytesRead, NULL); ` 1. Dokan would tunnel this down to the kernel mode driver and then back to the user mode file system code. It would be passed to a method like: NTSTATUS ProcessDeviceIoControl(LPCWSTR FileName, DWORD dwIoControlCode, LPVOID lpInBuffer, DWORD nInBufferSize, LPVOID lpOutBuffer, DWORD nOutBufferSize, LPDWORD lpBytesReturned, PDOKAN_FILE_INFO DokanFileInfo) 1. This method would resolve the dwIoControl code, perform the desired operation and return the requested bytes. Jim — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#544 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADHSOAGeAhK1Pk_DRJuyghXyG1i2prVOks5sN5vtgaJpZM4OKc5o> .

[jimuk]

The mechanism is generic. It can be used to pass any data in either direction from user app to user mode file system driver. It would provide a generic means to implement extension functionality without rebuilding Dolan each time.

Jim

[marinkobabic]

How we can prevent conflicts with existing implementation of Dokan? What happens when you send IOCTL for file writes or oplock? Am 14.07.2017 19:48 schrieb "jimuk" <[email protected]>:

…

The mechanism is generic. It can be used to pass any data in either direction from user app to user mode file system driver. It would provide a generic means to implement extension functionality without rebuilding Dolan each time. Jim — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#544 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADHSOHSnCeEYQvnlQzb95vSKJmqM8_6Bks5sN6n9gaJpZM4OKc5o> .

[jimuk]

I don't think there is any conflict. IOCTL is independent of normal data read/write. Typically it would be used for non-data requests lime retirving metadata but that is entirely up to implementtation. All Dorkan should do (like Windows) is pass the requests back and forth. It doesn't need to interpret

Jim

[marinkobabic]

If it works the way you explained then it would be a nice feature 😁 Am 14.07.2017 22:56 schrieb "jimuk" <[email protected]>:

…

I don't think there is any conflict. IOCTL is independent of normal data read/write. Typically it would be used for non-data requests lime retirving metadata but that is entirely up to implementtation. All Dorkan should do (like Windows) is pass the requests back and forth. It doesn't need to interpret Jim — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#544 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADHSOFJFTOG2_aPfW0Na_jc-qhq5JyaBks5sN9X0gaJpZM4OKc5o> .

[Maxhy]

At first sight it could be a nice feature as I personally already needed it to perform driver specific features.

But I'm asking myself about the following points, to discuss:

• we are in user mode, specific features could be implemented and used another way
• current trend is to make the user mode API simpler. The API doesn't have any IOCTL knowledge, this would a little bit kill the abstraction layer
• I'm concerned about isolation. Ideally Dokan will one day be able to handle and check all user-mode data before sending corrupted information to kernel components, to avoid BSOD. We will never be able to do that with custom IOCTL

[jimuk]

In response to @Maxhy's points I would make the following comments:

1. There may be other ways to communicate from user mode to the file system code but IOCTL is the standard way this is done in both Windows and Linux. It therefore represents the most universal way to provide this additional functionality.

2. IOCTL would not kill the abstraction layer. It provides a generic mechanism to "tunnel" application specific data over the abstraction layer. Windows uses the same model to "layer" driver and user mode code. Each Windows component can listen for specific IOCTL codes and, if appropriate, respond to them. In all other cases they are just passed down/up to the next layer. Again, this is therefore a universal approach. Similarly, Dokan doesn't need to understand the IOCTL packets to pass them up/down. The mechanism is truly generic.

3. Dokan will be able to validate the IOCTL packet buffer address and size. It can therefore check that the IOCTL mechanism is being called correctly and avoid BSOD issues in the driver. The only thing it won't be able to do is validate the contents of the IOCTL packet. However, this shouldn't be a concern this is an application specific detail that Dokan cannot be expected to know about.

[Rondom]

I think it is important to distinguish between custom IOCTLs/FSCTLs and Windows ones. The latter should probably be handled within the kernel driver itself, to keep things simple and keep some level of abstraction. What do you think?

How would one distinguish between the two? Is there a range that Windows reserves for third-party code? That way, we could basically say:

if (custom_ioctl)
    pass_to_userspace()
else {
    // handle windows  ioctls
}

[jimuk]

There really is nothing special about Windows IOCTL codes. They are relevant to specific Windows components but otherwise just like any other IOCTL to everything else. I would therefore suggest a slightly different approach:

a. Pass IOCTL codes through up/down the driver (e.g. in / out buffer)

b. Support specific IOCTL codes inside Dokan if relevant (for instance IOCTL_DISK_GET_DRIVE_GEOMETRY)

[atanamir]

I'd like to +1 this even though it's been inactive for some time. And reading through the whole thread, it's definitely a 2.0 feature because it would change the kernel <-> library implementation. We'd have to filter out the known (windows) FSCTLs/IOCTL codes in the kernel, but only pass unrecognized ones to the dokan user-mode process to complete.

In our case, we'd also like to use this for applications to request additional non-standard metadata from our dokan-based file system.

[jimuk]

There is no need to filter out known FSCTLS/IOCTLs. Windows uses an IO stack model which allows all unhandled control codes to filter down/up to the next layer - Basically, if you care you handle it. If you don't care / don't know you should ignore it.

I'm also not sure why this feature would need to wait to v2.0? It will require a change to the DOKAN_OPERATIONS structure but one additional callback will be sufficient to handle all control codes because the mechanism is generic and infinitely extensible.

Jim

[Liryna]

Changing the DOKAN_OPERATIONS structure would break the API, so we would have to change the major version.
https://semver.org/

[magol]

As @Liryna said, it is unfortunately not possible to extend functionality without breaking backward compatibility.

It's a problem, I'd love to see that we somehow change this model to version 2 so we can expand with new features in the future without breaking compability (se Open/closed principle). What it would look like, I do not know. Maybe create a new issue for that discusion?

[jimuk]

I understand that it would change the API and that this should best be done when introducing a new major version number. This would avoid any problems with backwards compatibility.

In the future, instead of using the DOKAN_OPERATIONS structure, a different solution would be to export a "registration" method from the DLL to register the callback/handler for each library operation such as read, write, IOCTL and so on. This would allow the interface to be extensible in the future and avoid the problems with the structure.

Jim

[magol]

@jimuk I agree with you. As long as the performance is not affected, I think it could be a good solution.

[Rondom]

A common pattern is to leave some reserved members at the end of the struct and instruct callers to keep them zero. Adding a new member does not break compatibility.

More extensible and also enabling providing ABI compatibility is using a version macro, i.e. users of the library define the version of the library that they desire. The header then defines the structs and the functions taking those structs according to the version. New versions still support the old version by converting the arguments accordingly. This is what I had planned to enable at least source-level compatibility for Dokan 1 => 2.

(I hope what I wrote is understandable.)

[kyanha]

I would like to +1 this. But I don't think the Windows fsctl codes should be filtered out -- one of the main reasons why we can't do a lot of the cool things that native filesystems can do is because we don't receive the Windows-defined fsctl codes (such things as "get reparse point", "set reparse point", "get file ID", "set file ID", "recall from remote storage", "encrypt file with EFS", "decrypt file with EFS", "zero a region of a sparse file", etc).

Being able to receive and dispatch those fsctls does also require being able to tell Windows that those capabilities are available to call from the filesystem.

[glenn-slayden]

I agree with @kyanha; I was initially excited to discover (and learn about) dokan(y), but I was assuming my custom filesystem would be able to emit NTFS reparse points, the small files that would need to be have a REPARSE_TAG attached (These would be simulated on-the-fly by my dokany FS to redirect the IO request back into an actual NTFS volume). It's a showstopper if I can't do this with your (otherwise entirely alluring) dokany project.

[Liryna]

I understand the purpose of this request and I am not against having it but this is a niche feature which I will personally not have time to implement.
The hope is someone that expect to be able to implement correctly the ioctl in user land would also have the knowledge to contribute to the project and add the feature on kernel side 🚀 😎

[kyanha]

One of the reasons folks use Dokany is because they're not set up for kernel-mode development. I might technically have the knowledge to be able to contribute to the kernel (even though I'm kind of terrified by the idea), but I don't have a setup which would allow me to do so without crashing my coding machine.

Are there good guides for this? Can I set up a test machibe in a VM? (I prefer using VirtualBox to Hyper-V, if that's a concern.)

[Liryna]

I agree that getting the setup is not that easy but it is documented here https://github.com/dokan-dev/dokany/wiki/Build https://github.com/dokan-dev/dokany/wiki/How-to-Debug-Dokan
If something is missing or someone have a question, feel free to ping me!

[kyanha]

I think it is important to distinguish between custom IOCTLs/FSCTLs and Windows ones. The latter should probably be handled within the kernel driver itself, to keep things simple and keep some level of abstraction. What do you think?

I disagree. You have no desire (or even possibility, since these depend intimately on the implementation) to implement (for example) the sparse file or compression FsCtls, but I would be quite happy to have the option to do so.

How would one distinguish between the two? Is there a range that Windows reserves for third-party code? That way, we could basically say:

if (custom_ioctl)
    pass_to_userspace()
else {
    // handle windows  ioctls
}

I would rather have the ability to implement the Windows Fsctls that you don't or can't. (There are a fair number of them, and trying to provide abstractions for all of them would balloon the DOKAN_OPERATIONS structure with necessary 00 entries for every implementation that didn't want to provide them -- which would/should be most of them.)

Instead, I'd rather Dokany do something like this:

switch (fsctlcode) {
case CREATE_LOCK:
// what Dokany currently does
break;
default:
returnval = dispatch_ioctl_to_usermode_or_return_invalid_operation_if_not_provided(fsctlargs...);
break;
}

If the DOKAN_OPERATIONS doesn't provide a valid function pointer for the Fsctl handler (or if the size of the DOKAN_OPERATIONS provided via the ioctl that is used to register the usermode is too short, which would allow for current filesystem implementations to work without needing to be recompiled), then it returns the invalid operation error code.

The expectation of the implementation is that it would switch on the fsctl code, process what it wants to process, and return Dokany's invalid operation value if it doesn't.

All an FsCtl is is a way for a program running in user mode to ask that the filesystem driver do something with the file handle that isn't specified in the normal create/read/update/delete/get limited metadata/set limited metadata API set. Stuff like "create symbolic or hard link", or "move this file to tape storage", or "set this range of bytes to \00 and reduce the actual storage used by the appropriate number of disk blocks if possible", or "give me the data's SHA-256 digest", or "present this JSON file as JSON-XML", or anything that makes sense to the programmer of the filesystem and the app that's using the filesystem.

All Dokany needs to do is handle what it wants to handle (which includes dispatching what it currently dispathes through its existing functions), and dispatch the rest through his (or fail the request if the dispatch can't happen). If Dokany wants to provide an abstraction for a Windows-defined filesystem function in the future, it could provide opt-in bits in the value passed to dokan_setup.

Or it can refactor in Dokany 2 or 3 to have a more complex structure to direct the kernel driver precisely on how it should act. I'm just currentky frustrated by not being able to have buffers passed to my implementation from my applications and back.

(The driver could even enforce that the caller and the implementation have different buffers, and copy the data between them, if it wants. The driver would need to allocate buffers in the implementation's memory space to do this, though, and I don't know the best way to do this in Windows.)

[kyanha]

As @Liryna said, it is unfortunately not possible to extend functionality without breaking backward compatibility.

Actually, it might be, though this is admittedly a bit of a hack. The size of the DOKAN_OPERATIONS structure can be calculated from the size of the buffer passed to the Ioctl that tells the Dokany driver to register the filesystem. Since the driver needs to copy it into its own memory, all it has to do is allocate its own dispatch table (which, since it's in the kernel, allocates and then zeroizes any allocation). Then copy
the calculated size of the passed DOKAN_OPERATIONS structure. Any new members have to be specified at the end of the structure anyway (there's no way to correctly change the interpretation of existing structure offsets), so after a short copy the remaining allocated memory would remain \00 bytes, which correspond to the value of NULL.

Then, the current "is this a valid pointer to call in the implementation" function would say it wasn't, and the current default "invalid operation" return would happen.

It's a problem, I'd love to see that we somehow change this model to version 2 so we can expand with new features in the future without breaking compability (se Open/closed principle). What it would look like, I do not know. Maybe create a new issue for that discusion?

[Liryna]

(The driver could even enforce that the caller and the implementation have different buffers, and copy the data between them, if it wants. The driver would need to allocate buffers in the implementation's memory space to do this, though, and I don't know the best way to do this in Windows.)

When I looked at removing the event that request userland larger buffer when write event cannot fit (here).
I saw that ZwAllocateVirtualMemory can commit allocated kernel memory to userspace meaning when the library try to pull something that does not fit the event buffer, we could allocate memory, attach it to the library process and point the event struct sent to userland to point to that buffer. On completion (or userland process die), we detach the buffer from the buffer and free it (the best would be to keep it in a buffer pool).

Actually, it might be, though this is admittedly a bit of a hack. The size of the DOKAN_OPERATIONS structure can be calculated from the size of the buffer passed to the Ioctl that tells the Dokany driver to register the filesystem....

Breaking the DOKAN_OPERATIONS (userland) isn't a big deal (we do it every minor version) compared to breaking kernel (as it requires reinstalling the driver) compatibility (EVENT_START or EVENT_INFORMATION) to say the truth.
Those two structs should end with a unused "large" buffer aligned to allow upgrade / changes.