pwendell

7 ms·

1.
by pwendell 1mo ago
We do this for a lot of our customers (fine tuned to save cost when inference volume is high). Right now for internal coding we are using off-the-shelf models but we are considering fine tuning as well to squeeze more efficiency out.
2.
by pwendell 1mo ago
Thank you for the feedback. We wrote this because after discussing with some of our peer companies, I realized everyone was roughly doing similar things. And I thought it would be good for someone to just systematically write down what thos
3.
by pwendell 1mo ago
The difficulty of evaluating coding agents is indeed a really big challenge. We built evals on our own codebase and shared some information about that to allow other companies to replicate. We found our own evals correlated loosely with pub
4.
by pwendell 1mo ago
The issue is the growth rates can cause costs to drastically change quickly. If you have 1,000 employees and the average is spending $100/month you're at a $1.2M run rate. But suddenly a new model comes out that's twice as ex
5.
by pwendell 1mo ago
I authored this - happy to answer any questions.
6.
Best-in-Class Open Source Generative Models(databricks.com)
2 points by pwendell 3y ago|0 comments
7.
by pwendell 3y ago
15k is the entire corpus we have right now. Hopefully others can join up in releasing additional samples that can be merged in over time. We'll definitely keep iterating on Dolly and releasing everything openly.
8.
by pwendell 3y ago
Yes this was a very surprising result... that the relatively small uptraining was able to unlock so much latent knowledge in the model.
9.
by pwendell 3y ago
Yes, this.
10.
by pwendell 3y ago
Yes the issue is that some of the training data is arguably tainted with some noncommercial license (it's nuanced, discussed below in my comment). We are releasing weights to people who request but we just wanted to have an email reque
11.
by pwendell 3y ago
No I haven't heard of that, we'll engage with that team. This is exactly what we need will look into it.
12.
by pwendell 3y ago
Yes it's nuanced, but will be simplified going forward. This uses a fully open source (liberally licensed) model and we also open sourced (liberally licensed) our own training code. However, the uptraining dataset of ~50,000 samples wa
13.
by pwendell 3y ago
Full source code is up here now: https://github.com/databrickslabs/dolly Sorry it took us a day to get the external repo setup.
14.
Announcing SparkR: R on Spark(databricks.com)
7 points by pwendell 11y ago|0 comments
15.
by pwendell 12y ago
The DataFrame is an evolution of the RDD model, where Spark knows explicit schema information. The core Spark RDD API is very generic and assumes nothing about the structure of the user's data. This is powerful, but ultimately the gene
16.
by pwendell 12y ago
Very likely in Spark 1.4. Hortonworks has been helping out with this, we just need some internal refacotring to the API to make it work.
17.
by pwendell 12y ago
No worries. Hopefully you'll reconsider using it!
18.
by pwendell 12y ago
Hey - sorry you had a bad experience. That bug was filed as a "minor" issue with only one user ever reporting it, so it didn't end up high up in our triage. We didn't merge the pull request because it was not correct, ho
19.
Managing Complexity(web.stanford.edu)
2 points by pwendell 12y ago|0 comments
20.
ML Pipelines: A New High-Level API for MLlib(databricks.com)
2 points by pwendell 12y ago|0 comments
21.
Announcing Spark 1.2(databricks.com)
2 points by pwendell 12y ago|0 comments
22.
A Scala serialization puzzle we ran into while porting Spark to Scala 2.11(gist.github.com)
5 points by pwendell 12y ago|0 comments
23.
by pwendell 12y ago
You can call GraphX algorithms right now from within the Streaming API, for instance compute a graph on a windowed view of data. Online graph algorithms aren't there yet (probably what you mean). We just started adding online MLlib alg
24.
by pwendell 12y ago
Yeah, you can load Parquet data directly into SchemaRDD's in 1.1 and get the type conversion, including use of nested types. That's the long term solution for all of our storage integration is to go through the SchemaRDD API since
25.
by pwendell 12y ago
Hey all - I'm the release manager for Spark 1.1. Happy to answer any questions about Spark or this release.
26.
Spark Summit streaming live all day(ustream.tv)
2 points by pwendell 13y ago|0 comments
27.
Sampling Twitter Using Declarative Streams(pwendell.com)
1 points by pwendell 13y ago|0 comments
28.
COS-597G: Surveillance and Countermeasures(cs.princeton.edu)
5 points by pwendell 13y ago|0 comments
29.
Free berkeley class this week on large scale machine learning(ampcamp.berkeley.edu)
4 points by pwendell 13y ago|0 comments
30.
Performance analysis of Hive, Redshift, Impala, and Shark(amplab.cs.berkeley.edu)
2 points by pwendell 13y ago|0 comments

More ›