Watching/notifications

[rektide]

A huge huge huge part of file-system management & capabilities as we use them is being notified about when something changes. Without this, awful awful awful terrible badly performing highly costly & deeply inadequate hacks grow up like weeds, all over the place, as bad terrible coders do an awful job of probing around to figure out "what has changed?". Huge amounts of engineering effort have gone into trying to tackle this issue. Works like node-watch (attempting to fill the painful dx gaps in node's fs.watch being non recursive) & watchman grow like weeds, consuming developer year after developer year of time to make maintain & sustain.

I love the works proposed here, DEARLY, and think they are so on target & in the right direction. But if we don't ship in 1.0 some kind of reactivity, some kind of file watching, given what I see on the rest of the web, it will literally never ever happen, and we will be very very very sad with the years of strife suffering misery & terrible problems that were so easily avoided had anyone bothered to stake in on caring about a moderately successful go at what file system apis need to be & do.

Please please please please please. Please make native-file-system api have watching capabilities. Please let me look at at a subdirectory tree, let me say, if anything in /foo/bar changes, let me know. Without this, the amount of polling & state checking will boil away the oceans, be the most tragic waste of our planet we could imagine. If this is included, we can blissfully begin to carefully slowly rebuild the web in a way that coherently integrates the user experience with the stored shared mutable state that is the filesystem. Being able to see & observe those mutations is a crucial capability that I would put much on the line (were I aware of any such means to) to bring forth to the web, to native-file-system, as it turns this from a dumb, dead, passive tool into something involved, dynamic, & reactive.

Please kindly include some kind of fs-watch like capability.

[pwnall]

Thank you for your feedback! Glad you care about this API! We also think it's a very important capability for the Web.

Watching isn't going to make it for 1.0. Implementing it would take a lot of work for a browser, so holding back 1.0 for watching would delay all other use cases that can benefit from what we have today. I understand that watching is very important to you, and it has some applications that I care about, but I wouldn't be fair to ask everyone else to wait while this happens.

If you're interested in making this happen, you could help by putting together a Chrome CL that shows how to implement watching well on Windows, macOS, and Linux. This would help assess how difficult it would be to implement this in a cross-platform browser -- my intuition is that it'd take ~2 engineering years, and I'd really like to be proven wrong.

[guest271314]

Please let me look at at a subdirectory tree, let me say, if anything in /foo/bar changes,

Does that definition of "changes" mean appending or removing a single character at, that is, any .txt file outside of Native File System methods being used to do so; for example a text editor changes a file that Native File System has permissions to read/write? Would changes to permission of the file performed at shell also be considered a change; e.g., a file permission being changed to read-only and/or write/execute during permission for Native File System being granted?

[rektide]

@pwnall The only thing that turns up when I search for "chrome cl" is this link to something in ChromiumOS, and there's not enough context on that page for me to understand enough to try to follow along. I don't see anything in the Developer Guide that seems very promising. I was thinking perhaps (https://chromium.googlesource.com/chromiumos/docs/+/master/developer_guide.md#Making-changes-to-packages-whose-source-code-is-checked-into-Chromium-OS-git-repositories) was going to give me something to chew on, but it doesn't seem to match up with anything in I see on the "create a cl" page? What was your ask again?

As for your ask for a cross-platform PoC, I have no access to Windows or Mac systems. I've also never worked on Chrome before. It probably would take me two man years to do, sounds about right.

I do want to help but this is also terrifying, and feels a little bit like being sent to never never land.

[rektide]

@guest271314 yes to all of the above. I am a bit of a file watching maximalist.

The filesystem is our shared source of truth in computing, and being able to observe the truth happening is important.

[guest271314]

@rektide A "a non-terminating procedure that happens to refer to itself" pattern which reads the file continuously? A copy of one or more files would need to be stored, correct? Consider the case of same file open in several tabs in a text editor, or multiple text editors referencing the same file; how to determine which file is the original? Timestamps, or is the original file not important, only indexes of the number of files open having the same path and name and differences between the files? How would the developer be able to process all changes to files in an event-based or Promise based pattern in such a case, that is, the same file potentially being changed multiple times based on input from multiple programs accessing the file? Have not tried reading read-only files in some time; what should occur when a file permission is set to read-only and permission to read the file metadata is requested?

[guest271314]

@rektide A DirectoryObserver and FileObserver, similar to PerformanceObserver, IntersectionObserver, and MutationObserver?

[guest271314]

Since the user must know the directories and files which potentially will be changed by a write procedure, the program can create a ReadableStream to read every byte of an instance of let stream = writer . asWritableStream(), or similar directory and file writing stream. During the write, notify the changes made to a copy of the original file, then delete the copy, the changed file taking the place of the original file reference.

Code to implement checking for changes made during writing to an existing file and notification code is not an issue.

Notification of changes to permissions of files could begin be creating a time range in which the notification should occur, to avoid a "a non-terminating procedure that happens to refer to itself" pattern relevant to asynchronous code running continuously, consuming computational resources, even when no permissions are being changed during every call to the procedure. For example, from now until then check if directory and/or file permissions changed. Or, a method that checks only when called, not continuously.

How many times should a notification occur? Once, when single instance of write/read procedure completes? Or should notification occur once for each and every change to underlying data source; and directory and file permissions?

[guest271314]

See

• https://stackoverflow.com/a/880278
• https://superuser.com/a/634313
• https://github.com/tartley/rerun2

[jespertheend]

Being able to listen for changes is an absolute necessity when making something like IDEs, photo/video editors or any web app that uses its own type of file browser. I'd love to see this added as well.

Most OSes include their own apis for watching for file changes, ReadDirectoryChangesW on windows for instance, or the File System Events API on MacOS.

I'm not too familiar with browser development so I can't really comment on how long it would take to implement something. But by having these apis to our disposal hopefully it should take a lot less time to implement this than when browser vendors have to implement their own file watching system.

[davidbarratt]

someone asked me if I could make https://github.com/onedrivejs/onedrive work in the web browser (i.e. no need do download or install anything at all), and you know what, it almost could, but what's missing is being able to watch a folder for changes, which is required for a syncing app like this one to work.

[Eloque]

someone asked me if I could make https://github.com/onedrivejs/onedrive work in the web browser (i.e. no need do download or install anything at all), and you know what, it almost could, but what's missing is being able to watch a folder for changes, which is required for a syncing app like this one to work.

I've managed to make a very simple monitoring solution by maintaining a list of files locally and then checking that against what I get back from the directory listing. Works pretty well. I run into not being able to store file handles or references. So, every time I start the page, I need to manually select the folder to monitor again. I am fine with having to ask for new permissons, but selecting it again and again is annoying.

[Goctionni]

I've been working on an app recently and the only reason I had to make it an electron app rather than a native web app, is the lack of a file system api (that includes watching). A watch functionality would be a really big deal, it would make native web-apps so much more powerful.

[jespertheend]

I just implemented a basic file watching system in my application. It's not as good as native events could be I think, but it gets the job done.
Basically I'm recursively iterating over all the files and checking if their modified date has been changed.
At first I was looping over the files as frequent as possible but this quickly sucks up lots of cpu. So instead I'm only checking for changes on the focus or visibilitychange events. That works a lot better and keeps the fans quiet.
I only have a few files though, not sure how well this would work with huge amounts.

[brainkim]

Watching isn't going to make it for 1.0. Implementing it would take a lot of work for a browser, so holding back 1.0 for watching would delay all other use cases that can benefit from what we have today.

Even if it isn’t implemented in browsers, getting a sketch of what a watch API would look like in the context of this proposal would be incredibly helpful for planning purposes!

[mkruisselbrink]

I wrote some early ideas of what this could look like in this doc. Of course once we actually get to work on this (not currently planned anytime soon) we might end up with something entirely different.

[amoscatelli]

Wow, this would be incredibly helpful. I have several projects what could become PWAs instead of heavy client applications just with this.

[brainkim]

@mkruisselbrink An Observer-like API would be incredible!

Although I always get a little antsy thinking about how all the DOM Observer APIs slightly vary. Like why does the FilesystemObserver’s observe() method return a promise instead of undefined like all the other observers. Also I think maybe the recursive option should be passed to observe() and not the constructor. And I am a little weirded out by the use of snake_case for FileSystemObserverEntry properties?

Thanks for working on this!

[mkruisselbrink]

Thanks for the feedback. The API in that doc is very much a strawman proposal with not much thought put in it yet. snake_case vs camelCase is definitely an oversight. Existing observer APIs seem inconsistent in if options are passed to the constructor or the observe method, although I guess most pass it to observe() so it would make sense to do that here. Having observe() return a promise allows a website to know when/if observation has actually started. And in general "this file can't be observed for whatever underlying platform reason" isn't something that can be determined synchronously. Although maybe we need a way to report errors/no longer being able to observe a file after observation started anyway. So not sure what the best API would be in this case.

[Pet3ris]

Building a browser-based debugger that would benefit significantly from this feature.

[tomayac]

One of our partners would be interested in being notified when a device like a portable USB harddisk or a USB pen (aka. USB stick) is plugged into the system.

[jimmywarting]

Bit of topic, but for now... Dose anyone have any half decent / hacky / ugly solution for watching for changes in files / directory that works as of today? it dosen't have to be fancy. I'm not gonna use it in production anyways. the folder is tiny, polling every 500-1000ms interval or so is fine by recursively scan the hole directory for something that's newer.

[jespertheend]

I've got a decent implementation over here: https://github.com/rendajs/Renda/blob/0bf109e31b7fdef6e4624be7610f87d7855e3725/editor/src/util/fileSystems/FsaEditorFileSystem.js
It's part of a larger project so not a drop in replacement or library that is easy to use. But it might be a good example of how an implementation could work.
The functions to look for are updateWatchTree and traverseWatchTree. Basically it reads a full directory recursively and compares the dates of any files. I reckon this is probably pretty slow for large directories, but it has worked well for me so far.

[tomayac]

You can maybe build upon what I do in OPFS Explorer, which is essentially polling the directory recursively and then naively “hashing” over the string representation: https://github.com/tomayac/opfs-explorer/blob/f86be0405bef6c68532be3a9fd347eab89e511a1/contentscript.js#L5-L48 and https://github.com/tomayac/opfs-explorer/blob/f86be0405bef6c68532be3a9fd347eab89e511a1/devtools.js#L145. Improvement could start at figuring out where in the tree the changes are to not rebuild it entirely. It worked OK so far in practice.

[jimmywarting]

I did kind of already started hacking together something before you even had the chance to suggest it to me...

Took a little bit of that and another little bit that and smashed it together to form a: a live reload 🎉
without any plugins or any specific websocket injections, extensions or server
Works with any editor / development setup

Just have to make it into a bookmarklet (maybe create some github repo - if folks want to find out about it and want to use it, suggest ideas and maybe improve it)
(tell me if i should)

Works by

• prompting the developer of a directory to watch
• saves the handle in indexedb so you can re-enable it again
• clears the current window and replaces it with a iframe
  (wanted to use portals, but there where no way to inject scripts (only send postMessages + they are still experimental)
• watch for unload events and reflect the url in the top window
• and then watch the directory for changes and then reloading the iframe

// https://github.com/WICG/file-system-access/issues/72#issuecomment-1426870564

async function watch(ms = 500) {
  let p
  // Short simple KV storage
  let query = (method, ...args) => (p ??= new Promise(rs => {
    const open = indexedDB.open('ymca-kv')
    open.onupgradeneeded = () => open.result.createObjectStore('kv')
    open.onsuccess = () => {
      const db = open.result
      query = (method, ...args) => {
        const q = db.transaction('kv', 'readwrite').objectStore('kv')[method](...args)
        return new Promise((rs, rj) => {
          q.onsuccess = () => rs(q.result)
          q.onerror = () => rj(q.error)
        })
      }
      rs()
    }
  })).then(() => query(method, ...args))

  const kv = (...args) => query(...args)
  
  // Create our own window that we can control / reload
  const iframe = document.createElement('iframe')
  iframe.src = location.href
  iframe.style = 'width: 100vw; height: 100vh; border: 0'
  iframe.onload = () => {
    history.replaceState(null, null, iframe.contentWindow.location.href)
    iframe.contentWindow.addEventListener('unload', () => {
      console.clear() // clear console on unload
      iframe.onload()
    })
  }
  
  // replace window
  document.documentElement.innerHTML = ''
  document.body.append(iframe)
  document.body.style = 'padding: 0; margin: 0'
  
  const root = await kv('get', 'root') || await showDirectoryPicker().then(async dir => {
    await kv('put', dir, 'root')
    return dir
  })
  
  const permission = await root.queryPermission()
  if (permission === 'prompt') await root.requestPermission({mode: 'read'})
  let lastChanged = Date.now()
  const sleep = () => new Promise(rs => setTimeout(rs, ms))
  
  // recursively find the last modified date of any file 
  async function findDate (source, bag = {mod: 0}) {
    if (source.kind === 'directory') {
      const entries = []
      for await (const entry of source.values()) {
        entries.push(entry)
      }
      await Promise.all(entries.map(entry => findDate(entry, bag)))
    } else {
      const mod = await source.getFile().then(file => file.lastModified)
      if (bag.mod < mod) bag.mod = mod
    }
    return bag.mod
  }

  // start the watcher
  while (1) {
    await sleep(ms)
    const newest = await findDate(root)
    if (newest > lastChanged) {
      lastChanged = newest
      iframe.contentWindow.location.reload()
    }
  }
}

watch(500)

[josephrocca]

Just saw this tweet and fitured I'd drop Deno's watchFs here for potential inspiration on API.

const watcher = Deno.watchFs("/");
for await (const event of watcher) {
   console.log(">>>> event", event);
   // { kind: "create", paths: [ "/foo.txt" ] }
}

So it's an async iterable and you can call watcher.close() to stop. Seems quite nice. But might also be good to allow for a simple callback approach too, otherwise, IIUC, you'd need to always wrap it in a function if you don't want it to await, which seems a bit messy.

I do like how little "indirection" there is with the async iterable approach - feels very natural/readable. Might just be me though.

Linking to the API proposed earlier in this thread by mkruisselbrink for reference/comparison: https://docs.google.com/document/d/1jYXOZGen4z7kNrKnwBk5z4tbGRmGXmQ9nmoyJRm-V9M/edit?usp=sharing

async function change_listener(changes, observer) {
  // do something with changes
}

let observer = new FileSystemObserver(change_listener);
observer.observe(file_handle);
observer.observe(directory_handle, {recursive: true});

// …

observer.disconnect();

(Could observer be async iterable?)

[jespertheend]

To be honest I'm not particularly fond of Deno's watch api.
I feel like async iterators are meant for iterating over array like objects, not for events.
Using addEventListener or something like an observer feels much more web-y to me.

[jsejcksn]

@jespertheend I didn't interpret @josephrocca's suggestion to mean exclusion of an observer/subscription pattern — rather an object which implements both interfaces simultaneously to provide the user with greater API flexibility.

On topic: @domenic provided a juxtaposition of those patterns in this answer on Stack Overflow. In the answer, he remarks:

It's worth noting that you can build either approach on top of the other in a pinch:

• To build push on top of pull, constantly be pulling from the pull API, and then push out the chunks to any consumers.
• To build pull on top of push, subscribe to the push API immediately, create a buffer that accumulates all results, and when someone pulls, grab it from that buffer. (Or wait until the buffer becomes non-empty, if your consumer is pulling faster than the wrapped push API is pushing.)

The latter is generally much more code to write than the former.

[jespertheend]

Ah sorry I misread the suggestion. Yeah having both options available would be nice.

this answer on Stack Overflow

Interesting! The whole answer give some good insights on this exact situation.

[a-sully]

A proposal for a new FileSystemObserver interface is at whatwg/fs#124