7 ms·

Indeed, the space is growing quickly. I run a company that provides a backend for real-time collaboration, so we necessarily stay on top of what's out there.

Indeed, the space is growing quickly. I run a company that provides a backend for real-time collaboration, so we necessarily stay on top of what's out there. There's a reason the server-side tech tends be paid: it is extraordinarily difficult to provide guaranteed eventual consistency of data at low latency. The CKEditor guys (and us, for that matter) have put YEARS of development effort into their solutions. We're working on both offline mode and a generic rich-text data model with support for the major editors. If anyone's interested, I'd happily do some Q and A here.

blotter_paper 7y ago
I appreciate that you mentioned a competitor without shamelessly plugging your own product, but out of curiosity, and if you don't mind sharing, what editor do you work on?
- alalonde 7y ago
  You can currently already do good-enough-for-most DOM-level synchronization using any editor, of which we have a few examples. A lot of the examples of RTC out there (apps using TogetherJS for instance) are doing this already. For rich text, we'd like to be able to provide 100% support for a particular editor's capabilities, which necessitates a deeper, custom data model. One of our value propositions is a unified backend for dealing with any JSON data (and eventually beyond), avoiding binding server-side code to a particular choice of UI component.
  - houshuang 7y ago
    I am curious about this argument - I think CKEditor 5 made a similar one in this epic blog post about how they implemented real-time collaboration (https://ckeditor.com/blog/Lessons-learned-from-creating-a-rich-text-editor-with-real-time-collaboration/ https://ckeditor.com/blog/Lessons-learned-from-creating-a-ri...). We're using Quill.js with ShareDB, which supports JSON structures (which is great, because for us we often have documents with several rich text fields, and other complex structures). So far we've been able to do anything we wanted with Quill, and I've never felt limited by the data structures we have available... (We also do all kinds of other stuff with ShareDB JSON). I guess one reason you could need custom types would be to ensure consistency - if two keys depend on each other, and one user sets one key, and the other user sets another key, and the document is now invalid, you'd need the engine to be able to reconcile at a higher level?
    - scofalik 7y ago
      Author of the linked blog post here. The example cases for additional types / custom implementation are in this section: https://ckeditor.com/blog/Lessons-learned-from-creating-a-rich-text-editor-with-real-time-collaboration/#advanced-conflict-resolution https://ckeditor.com/blog/Lessons-learned-from-creating-a-ri... These content-preservation edge cases weren't possible to solve with what was available (at least at the time we started the project). Even apart of that, ottypes/json0 was lacking some basic things, like moving objects. I see they came up with a new implementation recently (https://github.com/ottypes/json1 https://github.com/ottypes/json1) and it allows moving objects. Maybe the new implementation would solve some problems. However, it is in "preview" state, and the last update was 2 months ago, so I am not sure how well it will be maintained. Also, there are some edge cases when transforming ranges (which CKE5 use to represent, for example, comments on text or content created in track changes mode). I don't want to bury you in difficult to understand examples but if you are interested you might want to check the examples listed in inline codes for this function: https://github.com/ckeditor/ckeditor5-engine/blob/master/src/model/range.js#L574 https://github.com/ckeditor/ckeditor5-engine/blob/master/src.... As far as Quill.js is concerned, it is based on the linear data model, which brings limitations when it comes to complex features. Transformation algorithms for linear data models are much simpler and there are more implementations and articles in this area. Everything depends on your needs. If Quill.js features set and functionality fit your needs then the solution you chose is correct. With CKE5 however, we didn't want to go on any compromises. We needed complex structures for our features, and for having a powerful framework - to enable other developers to write whatever feature they want and have those features working in real-time collaboration. We wanted transformation algorithms which will handle all the edge cases. It is true, some of those cases are quite rare. And the old "10/90" mantra applies here, in this case "10% of use cases brings 90% of complexity". But those edge cases happen and we didn't want to disappoint our users.
    - scofalik 7y ago
      > (...) if two keys depend on each other, and one user sets one key, and the other user sets another key, and the document is now invalid, you'd need the engine to be able to reconcile at a higher level? I am not sure if I understand you correctly here, but it's not really that. Could you give me a more concrete example? The kind of problems for extra types are, for example: user A changes a paragraph to a list item and user B splits it. As a result you'd like to have two list items instead of a list item followed by a paragraph. This is impossible if you don't give more semantic meaning to the operations. There are other problems though, as you mentioned - with invalid document. For example, you have this kind of a list: * Foo __* Bar __* Baz User A outdents "Bar" and user B indents "Baz" creating a list like this: * Foo * Bar ____* Baz In CKE5 this is an incorrect list (we don't allow indent differences bigger than one). This cannot be fixed through OT so we fix it in post-fixers which are fired after all the changes are applied.
  - aloer 7y ago
    Could you point us to some of those togetherJS apps? I just recently learned about it in the context of collaborative browsing. How is this related to collaborative text stuff?
    - alalonde 7y ago
      If you scroll down a bit there are a few examples, including jsfiddle: https://togetherjs.com/ https://togetherjs.com/ Essentially, they are doing OT on the DOM.
- mmacfadden 7y ago
  I assume you found us already, but more info can be found at https://convergence.io https://convergence.io. There are some examples that demonstrate a few of the editors as https://examples.convergence.io/ https://examples.convergence.io/
pmontra 7y ago
Are you working only with editors running inside web browsers or also native ones? What I would like to have is something that bridges two native editors, let's say emacs and VS Code, and let's a team share the editor (whatever each developer's preferred editor is) and not only the screen on Slack or similar. That would be great for remote pair programming. I guess there would be a protocol for selecting the buffer/tab and opening files (with local approval to prevent remote file reads/writes).
- unhammer 7y ago
  https://floobits.com/ https://floobits.com/ at least claims to support Emacs and Atom (and neovim and sublime and intellij); though https://github.com/Floobits/floobits-vscode https://github.com/Floobits/floobits-vscode says "Not ready. Don't get your hopes up."
- lloeki 7y ago
  > What I would like to have is something that bridges two native editors The big failing of co-editors is that currently they all silo into a single editor. THat's OK for e.g co-editing stuff on a web platform (e.g Google Docs) but it totally sucks for collaborating on code/documents you edit locally, forcing everyone to use the same editor/IDE, which ultimately fails because no one will use the same editor/IDE across a whole company/team/project, so although useful I've never seen one take hold for anything but a very short amount of time in any company. We desperately need (a couple of) well-defined protocols that get implemented across editors, and one to emerge as a widely supported winner, whatever its shortcomings.
  - lilmoelil 7y ago
    I think 'Google Wave Operational Transformation' was an attempt at this. I believe there is some merit in this; though my doubts come from the failure of Google Wave.
  - mmacfadden 7y ago
    For plain text, this is not so hard. The data model for plain text (e.g. a string) and the set of mutations on that model are pretty small. Also describing where a cursor is and what is selected is likewise fairly straight forward. Co-editing between plain text editors is completely doable, IMHO. It's much harder for rich text editors (RTEs) because the various RTE's vary widely in the exact subset of rich text features they support. One will support tables, and another will not. One will support video and another will not. One will use a linear position to describe where the cursor is, and another will use a DOM Range. This makes it very hard to support co-editing between different rich text editors. It's not impossible, just hard enough where most of us are not tackling it.
gtm1260 7y ago
What kind of data structures are essential to know to work on these kind of projects?
- kmelve 7y ago
  We (at sanity.io) get the work done with https://www.portabletext.org https://www.portabletext.org.
  - manigandham 7y ago
    There's also MobileDoc: https://github.com/bustle/mobiledoc-kit/blob/master/MOBILEDOC.md https://github.com/bustle/mobiledoc-kit/blob/master/MOBILEDO...
tjungblut 7y ago
Do you guys rely on a central authority approach like Prosemirror does? I'm desperately looking for something that's somewhat stateless or at least replicable (even better if shardable) on a document basis.
- mmacfadden 7y ago
  We do have a server in the architecture, and there is a small amount of state maintained by the server, per document as it is being edited. However the system is clusterable and documents shard across the cluster.
houshuang 7y ago
Thanks for the component you provided for shared cursors in text areas (https://github.com/convergencelabs/html-text-collab-ext https://github.com/convergencelabs/html-text-collab-ext). I've been using it with ShareDB, and have been working on adapting it to work with single text inputs, which has been surprisingly difficult (mainly dealing with scroll/overflow behaviour).
- mmacfadden 7y ago
  Our pleasure! If you want to share what you are doing we might be able to incorporate it into the utility for you! Happy to support a pull request, or to work collaboratively on it.

blotter_paper 7y ago

I appreciate that you mentioned a competitor without shamelessly plugging your own product, but out of curiosity, and if you don't mind sharing, what editor do you work on?

alalonde 7y ago
You can currently already do good-enough-for-most DOM-level synchronization using any editor, of which we have a few examples. A lot of the examples of RTC out there (apps using TogetherJS for instance) are doing this already. For rich text, we'd like to be able to provide 100% support for a particular editor's capabilities, which necessitates a deeper, custom data model. One of our value propositions is a unified backend for dealing with any JSON data (and eventually beyond), avoiding binding server-side code to a particular choice of UI component.
- houshuang 7y ago
  I am curious about this argument - I think CKEditor 5 made a similar one in this epic blog post about how they implemented real-time collaboration (https://ckeditor.com/blog/Lessons-learned-from-creating-a-rich-text-editor-with-real-time-collaboration/ https://ckeditor.com/blog/Lessons-learned-from-creating-a-ri...). We're using Quill.js with ShareDB, which supports JSON structures (which is great, because for us we often have documents with several rich text fields, and other complex structures). So far we've been able to do anything we wanted with Quill, and I've never felt limited by the data structures we have available... (We also do all kinds of other stuff with ShareDB JSON). I guess one reason you could need custom types would be to ensure consistency - if two keys depend on each other, and one user sets one key, and the other user sets another key, and the document is now invalid, you'd need the engine to be able to reconcile at a higher level?
  - scofalik 7y ago
    Author of the linked blog post here. The example cases for additional types / custom implementation are in this section: https://ckeditor.com/blog/Lessons-learned-from-creating-a-rich-text-editor-with-real-time-collaboration/#advanced-conflict-resolution https://ckeditor.com/blog/Lessons-learned-from-creating-a-ri... These content-preservation edge cases weren't possible to solve with what was available (at least at the time we started the project). Even apart of that, ottypes/json0 was lacking some basic things, like moving objects. I see they came up with a new implementation recently (https://github.com/ottypes/json1 https://github.com/ottypes/json1) and it allows moving objects. Maybe the new implementation would solve some problems. However, it is in "preview" state, and the last update was 2 months ago, so I am not sure how well it will be maintained. Also, there are some edge cases when transforming ranges (which CKE5 use to represent, for example, comments on text or content created in track changes mode). I don't want to bury you in difficult to understand examples but if you are interested you might want to check the examples listed in inline codes for this function: https://github.com/ckeditor/ckeditor5-engine/blob/master/src/model/range.js#L574 https://github.com/ckeditor/ckeditor5-engine/blob/master/src.... As far as Quill.js is concerned, it is based on the linear data model, which brings limitations when it comes to complex features. Transformation algorithms for linear data models are much simpler and there are more implementations and articles in this area. Everything depends on your needs. If Quill.js features set and functionality fit your needs then the solution you chose is correct. With CKE5 however, we didn't want to go on any compromises. We needed complex structures for our features, and for having a powerful framework - to enable other developers to write whatever feature they want and have those features working in real-time collaboration. We wanted transformation algorithms which will handle all the edge cases. It is true, some of those cases are quite rare. And the old "10/90" mantra applies here, in this case "10% of use cases brings 90% of complexity". But those edge cases happen and we didn't want to disappoint our users.
  - scofalik 7y ago
    > (...) if two keys depend on each other, and one user sets one key, and the other user sets another key, and the document is now invalid, you'd need the engine to be able to reconcile at a higher level? I am not sure if I understand you correctly here, but it's not really that. Could you give me a more concrete example? The kind of problems for extra types are, for example: user A changes a paragraph to a list item and user B splits it. As a result you'd like to have two list items instead of a list item followed by a paragraph. This is impossible if you don't give more semantic meaning to the operations. There are other problems though, as you mentioned - with invalid document. For example, you have this kind of a list: * Foo __* Bar __* Baz User A outdents "Bar" and user B indents "Baz" creating a list like this: * Foo * Bar ____* Baz In CKE5 this is an incorrect list (we don't allow indent differences bigger than one). This cannot be fixed through OT so we fix it in post-fixers which are fired after all the changes are applied.
- aloer 7y ago
  Could you point us to some of those togetherJS apps? I just recently learned about it in the context of collaborative browsing. How is this related to collaborative text stuff?
  - alalonde 7y ago
    If you scroll down a bit there are a few examples, including jsfiddle: https://togetherjs.com/ https://togetherjs.com/ Essentially, they are doing OT on the DOM.
mmacfadden 7y ago
I assume you found us already, but more info can be found at https://convergence.io https://convergence.io. There are some examples that demonstrate a few of the editors as https://examples.convergence.io/ https://examples.convergence.io/

pmontra 7y ago

Are you working only with editors running inside web browsers or also native ones? What I would like to have is something that bridges two native editors, let's say emacs and VS Code, and let's a team share the editor (whatever each developer's preferred editor is) and not only the screen on Slack or similar. That would be great for remote pair programming. I guess there would be a protocol for selecting the buffer/tab and opening files (with local approval to prevent remote file reads/writes).

unhammer 7y ago
https://floobits.com/ https://floobits.com/ at least claims to support Emacs and Atom (and neovim and sublime and intellij); though https://github.com/Floobits/floobits-vscode https://github.com/Floobits/floobits-vscode says "Not ready. Don't get your hopes up."
lloeki 7y ago
> What I would like to have is something that bridges two native editors The big failing of co-editors is that currently they all silo into a single editor. THat's OK for e.g co-editing stuff on a web platform (e.g Google Docs) but it totally sucks for collaborating on code/documents you edit locally, forcing everyone to use the same editor/IDE, which ultimately fails because no one will use the same editor/IDE across a whole company/team/project, so although useful I've never seen one take hold for anything but a very short amount of time in any company. We desperately need (a couple of) well-defined protocols that get implemented across editors, and one to emerge as a widely supported winner, whatever its shortcomings.
- lilmoelil 7y ago
  I think 'Google Wave Operational Transformation' was an attempt at this. I believe there is some merit in this; though my doubts come from the failure of Google Wave.
- mmacfadden 7y ago
  For plain text, this is not so hard. The data model for plain text (e.g. a string) and the set of mutations on that model are pretty small. Also describing where a cursor is and what is selected is likewise fairly straight forward. Co-editing between plain text editors is completely doable, IMHO. It's much harder for rich text editors (RTEs) because the various RTE's vary widely in the exact subset of rich text features they support. One will support tables, and another will not. One will support video and another will not. One will use a linear position to describe where the cursor is, and another will use a DOM Range. This makes it very hard to support co-editing between different rich text editors. It's not impossible, just hard enough where most of us are not tackling it.

gtm1260 7y ago

What kind of data structures are essential to know to work on these kind of projects?

kmelve 7y ago
We (at sanity.io) get the work done with https://www.portabletext.org https://www.portabletext.org.
- manigandham 7y ago
  There's also MobileDoc: https://github.com/bustle/mobiledoc-kit/blob/master/MOBILEDOC.md https://github.com/bustle/mobiledoc-kit/blob/master/MOBILEDO...

tjungblut 7y ago

Do you guys rely on a central authority approach like Prosemirror does? I'm desperately looking for something that's somewhat stateless or at least replicable (even better if shardable) on a document basis.

mmacfadden 7y ago
We do have a server in the architecture, and there is a small amount of state maintained by the server, per document as it is being edited. However the system is clusterable and documents shard across the cluster.

houshuang 7y ago

Thanks for the component you provided for shared cursors in text areas (https://github.com/convergencelabs/html-text-collab-ext https://github.com/convergencelabs/html-text-collab-ext). I've been using it with ShareDB, and have been working on adapting it to work with single text inputs, which has been surprisingly difficult (mainly dealing with scroll/overflow behaviour).

mmacfadden 7y ago
Our pleasure! If you want to share what you are doing we might be able to incorporate it into the utility for you! Happy to support a pull request, or to work collaboratively on it.