Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make individual colors from Vega color schemes accessible from Altair #2617

Open
1 task
Tracked by #2861
joelostblom opened this issue Jun 15, 2022 · 18 comments
Open
1 task
Tracked by #2861
Labels
enhancement vega: vega Requires upstream action in `vega`

Comments

@joelostblom
Copy link
Contributor

joelostblom commented Jun 15, 2022

Tasks

Description

I think it would be valuable to make all the individual colors from Vega schemes accessible programmatically from Altair. Since Vega stores them in a single file, we could parse that and create a Python dictionary that could be used to access single colors from each color scheme. Something like this:

Code block

pd.read_table(
    'https://raw.githubusercontent.com/vega/vega/v5.21.0/packages/vega-scale/src/palettes.js',
    skipinitialspace=True,
    sep=':',
).drop(
    index=['export const discrete = {', '};']
).iloc[:, 0].str.replace(
    "'", ""
).str.replace(
    ',', ''
).apply(
    lambda x: ["#" + x[i:i+6] for i in range(0, len(x), 6)]
).to_dict()
{'blues': [
  '#cfe1f2',
  '#bed8ec',
  '#a8cee5',
  '#8fc1de',
  '#74b2d7',
  '#5ba3cf',
  '#4592c6',
  '#3181bd',
  '#206fb2',
  '#125ca4',
  '#0a4a90'],
 'greens': [
  '#d3eecd',
  '#c0e6ba',
  '#abdda5',
  '#94d391',
  '#7bc77d',
  '#60ba6c',
  '#46ab5e',
  '#329a51',
  '#208943',
  '#0e7735',
  '#036429']
...

@dangotbanned
Copy link
Member

dangotbanned commented Jan 3, 2025

@joelostblom I'm adding some context for what is going on in your script, and linking docs for Vega/Color Schemes

The lambda appears to be similar to d3-scale-chromatic/src/colors.js, with an example of usage in category10.js.
In short, splitting a string containing multiple hexadecimal colors

src/categorical/category10.js

https://github.com/d3/d3-scale-chromatic/blob/2c52792197299346b7bdb94322bb4dff8f554fea/src/categorical/category10.js#L3

A related comment by @domoritz (vega/vega#3843 (comment))

import colors from "../colors.js";

export default colors("1f77b4ff7f0e2ca02cd627289467bd8c564be377c27f7f7fbcbd2217becf");

The current palettes.js has changed since this issue was opened.
Lines L1-L12 and L78-L94 would probably break the initial parser idea.

I do agree these would be nice to have available in altair.
I think we should try to push for a Public API for this upstream in (https://github.com/vega/vega) - since that would be safer than dependending on internal library code.
That route was really helpful for me when I started reaching for vl_convert internals in vega/vl-convert#191

@dangotbanned dangotbanned added the vega: vega Requires upstream action in `vega` label Jan 7, 2025
@joelostblom
Copy link
Contributor Author

I think we should try to push for a Public API for this upstream in (https://github.com/vega/vega) - since that would be safer than dependending on internal library code.

Yes, I agree that this is the better option

@dangotbanned
Copy link
Member

@joelostblom something that might help push this forward is an example that would benefit from more granular access to the color palettes.

It could be an existing altair example, one from an upstream gallery that is difficult to reproduce or something entirely new.

Maybe even adapting some of the code from @mattijn's work in doc/user_guide/color_usage.rst.
I imagine being able to simplify our docs with help from upstream would be a net positive - and hopefully a step towards merging #3021

@dangotbanned
Copy link
Member

cc @hydrosquall regarding #2617 (comment)

Wondering if you had any thoughts how we could expose the colors within a scheme from vega?

I feel like this would be simpler than vega/vega#1938

@hydrosquall
Copy link
Member

I think vega/vega#1938 might not help with this issue, since Vega expressions do not have knowledge of color palettes.

A naive Altair question as I haven't changed it before - do you have the ability to Altair's Python code ask Vega JS runtime for a response, so Vega can be an input to Python rather than just an output? If you can't do it at runtime, you could write a json file into Altair in a CI step.

The schemes are queryable through the same API used to add new schemes, so you wouldn't have to do parsing of a JS file from a URL (though that approach is very ingenious).

https://vega.github.io/vega/docs/api/extensibility/#schemes

@dangotbanned
Copy link
Member

Clarification

I think vega/vega#1938 might not help with this issue, since Vega expressions do not have knowledge of color palettes.

Apologies @hydrosquall, the link I was drawing between these issues wasn't very clear (my bad).

What they have in common is that resolving either could result in new .json describing some currently underspecified functionality.

So if that was the goal here, I think we'd be able to get there with less effort than (vega/vega#1938).
As you've mentioned in (#2617 (comment)), we could evaluate this within the existing Vega JS runtime.

Re altair

A naive Altair question as I haven't changed it before - do you have the ability to Altair's Python code ask Vega JS runtime for a response, so Vega can be an input to Python rather than just an output?
If you can't do it at runtime, you could write a json file into Altair in a CI step.

I don't think we can interact with the Vega JS runtime natively (related #3365).
It is possible with additional (pretty heavy) dependencies (e.g. #3630, docs) - but I'd prefer pre-evaluating here so the colors are available for all users.

Right now, we have the schemes by name only:

altair.vegalite.v5.schema._typing.ColorScheme_T

ColorScheme_T: TypeAlias = Literal[
"accent",
"category10",
"category20",
"category20b",
"category20c",
"dark2",
"paired",
"pastel1",
"pastel2",
"set1",
"set2",
"set3",
"tableau10",
"tableau20",
"observable10",
"blues",
"tealblues",
"teals",
"greens",
"browns",
"greys",
"purples",
"warmgreys",
"reds",
"oranges",
"turbo",
"viridis",
"inferno",
"magma",
"plasma",
"cividis",
"bluegreen",
"bluegreen-3",
"bluegreen-4",
"bluegreen-5",
"bluegreen-6",
"bluegreen-7",
"bluegreen-8",
"bluegreen-9",
"bluepurple",
"bluepurple-3",
"bluepurple-4",
"bluepurple-5",
"bluepurple-6",
"bluepurple-7",
"bluepurple-8",
"bluepurple-9",
"goldgreen",
"goldgreen-3",
"goldgreen-4",
"goldgreen-5",
"goldgreen-6",
"goldgreen-7",
"goldgreen-8",
"goldgreen-9",
"goldorange",
"goldorange-3",
"goldorange-4",
"goldorange-5",
"goldorange-6",
"goldorange-7",
"goldorange-8",
"goldorange-9",
"goldred",
"goldred-3",
"goldred-4",
"goldred-5",
"goldred-6",
"goldred-7",
"goldred-8",
"goldred-9",
"greenblue",
"greenblue-3",
"greenblue-4",
"greenblue-5",
"greenblue-6",
"greenblue-7",
"greenblue-8",
"greenblue-9",
"orangered",
"orangered-3",
"orangered-4",
"orangered-5",
"orangered-6",
"orangered-7",
"orangered-8",
"orangered-9",
"purplebluegreen",
"purplebluegreen-3",
"purplebluegreen-4",
"purplebluegreen-5",
"purplebluegreen-6",
"purplebluegreen-7",
"purplebluegreen-8",
"purplebluegreen-9",
"purpleblue",
"purpleblue-3",
"purpleblue-4",
"purpleblue-5",
"purpleblue-6",
"purpleblue-7",
"purpleblue-8",
"purpleblue-9",
"purplered",
"purplered-3",
"purplered-4",
"purplered-5",
"purplered-6",
"purplered-7",
"purplered-8",
"purplered-9",
"redpurple",
"redpurple-3",
"redpurple-4",
"redpurple-5",
"redpurple-6",
"redpurple-7",
"redpurple-8",
"redpurple-9",
"yellowgreenblue",
"yellowgreenblue-3",
"yellowgreenblue-4",
"yellowgreenblue-5",
"yellowgreenblue-6",
"yellowgreenblue-7",
"yellowgreenblue-8",
"yellowgreenblue-9",
"yellowgreen",
"yellowgreen-3",
"yellowgreen-4",
"yellowgreen-5",
"yellowgreen-6",
"yellowgreen-7",
"yellowgreen-8",
"yellowgreen-9",
"yelloworangebrown",
"yelloworangebrown-3",
"yelloworangebrown-4",
"yelloworangebrown-5",
"yelloworangebrown-6",
"yelloworangebrown-7",
"yelloworangebrown-8",
"yelloworangebrown-9",
"yelloworangered",
"yelloworangered-3",
"yelloworangered-4",
"yelloworangered-5",
"yelloworangered-6",
"yelloworangered-7",
"yelloworangered-8",
"yelloworangered-9",
"darkblue",
"darkblue-3",
"darkblue-4",
"darkblue-5",
"darkblue-6",
"darkblue-7",
"darkblue-8",
"darkblue-9",
"darkgold",
"darkgold-3",
"darkgold-4",
"darkgold-5",
"darkgold-6",
"darkgold-7",
"darkgold-8",
"darkgold-9",
"darkgreen",
"darkgreen-3",
"darkgreen-4",
"darkgreen-5",
"darkgreen-6",
"darkgreen-7",
"darkgreen-8",
"darkgreen-9",
"darkmulti",
"darkmulti-3",
"darkmulti-4",
"darkmulti-5",
"darkmulti-6",
"darkmulti-7",
"darkmulti-8",
"darkmulti-9",
"darkred",
"darkred-3",
"darkred-4",
"darkred-5",
"darkred-6",
"darkred-7",
"darkred-8",
"darkred-9",
"lightgreyred",
"lightgreyred-3",
"lightgreyred-4",
"lightgreyred-5",
"lightgreyred-6",
"lightgreyred-7",
"lightgreyred-8",
"lightgreyred-9",
"lightgreyteal",
"lightgreyteal-3",
"lightgreyteal-4",
"lightgreyteal-5",
"lightgreyteal-6",
"lightgreyteal-7",
"lightgreyteal-8",
"lightgreyteal-9",
"lightmulti",
"lightmulti-3",
"lightmulti-4",
"lightmulti-5",
"lightmulti-6",
"lightmulti-7",
"lightmulti-8",
"lightmulti-9",
"lightorange",
"lightorange-3",
"lightorange-4",
"lightorange-5",
"lightorange-6",
"lightorange-7",
"lightorange-8",
"lightorange-9",
"lighttealblue",
"lighttealblue-3",
"lighttealblue-4",
"lighttealblue-5",
"lighttealblue-6",
"lighttealblue-7",
"lighttealblue-8",
"lighttealblue-9",
"blueorange",
"blueorange-3",
"blueorange-4",
"blueorange-5",
"blueorange-6",
"blueorange-7",
"blueorange-8",
"blueorange-9",
"blueorange-10",
"blueorange-11",
"brownbluegreen",
"brownbluegreen-3",
"brownbluegreen-4",
"brownbluegreen-5",
"brownbluegreen-6",
"brownbluegreen-7",
"brownbluegreen-8",
"brownbluegreen-9",
"brownbluegreen-10",
"brownbluegreen-11",
"purplegreen",
"purplegreen-3",
"purplegreen-4",
"purplegreen-5",
"purplegreen-6",
"purplegreen-7",
"purplegreen-8",
"purplegreen-9",
"purplegreen-10",
"purplegreen-11",
"pinkyellowgreen",
"pinkyellowgreen-3",
"pinkyellowgreen-4",
"pinkyellowgreen-5",
"pinkyellowgreen-6",
"pinkyellowgreen-7",
"pinkyellowgreen-8",
"pinkyellowgreen-9",
"pinkyellowgreen-10",
"pinkyellowgreen-11",
"purpleorange",
"purpleorange-3",
"purpleorange-4",
"purpleorange-5",
"purpleorange-6",
"purpleorange-7",
"purpleorange-8",
"purpleorange-9",
"purpleorange-10",
"purpleorange-11",
"redblue",
"redblue-3",
"redblue-4",
"redblue-5",
"redblue-6",
"redblue-7",
"redblue-8",
"redblue-9",
"redblue-10",
"redblue-11",
"redgrey",
"redgrey-3",
"redgrey-4",
"redgrey-5",
"redgrey-6",
"redgrey-7",
"redgrey-8",
"redgrey-9",
"redgrey-10",
"redgrey-11",
"redyellowblue",
"redyellowblue-3",
"redyellowblue-4",
"redyellowblue-5",
"redyellowblue-6",
"redyellowblue-7",
"redyellowblue-8",
"redyellowblue-9",
"redyellowblue-10",
"redyellowblue-11",
"redyellowgreen",
"redyellowgreen-3",
"redyellowgreen-4",
"redyellowgreen-5",
"redyellowgreen-6",
"redyellowgreen-7",
"redyellowgreen-8",
"redyellowgreen-9",
"redyellowgreen-10",
"redyellowgreen-11",
"spectral",
"spectral-3",
"spectral-4",
"spectral-5",
"spectral-6",
"spectral-7",
"spectral-8",
"spectral-9",
"spectral-10",
"spectral-11",
"rainbow",
"sinebow",
]

This is still helpful, but say for example "bluegreen" we can only know through trial-and-error what was encoded at each stop:

"bluegreen",
"bluegreen-3",
"bluegreen-4",
"bluegreen-5",
"bluegreen-6",
"bluegreen-7",
"bluegreen-8",
"bluegreen-9",

vega != vega-lite

I hadn't noticed before writing this, but we have 300+ color scheme names and that is a huge jump from https://vega.github.io/vega/docs/schemes/
Leads me to think the logic for this would need to be based on vega-lite, rather than vega.

Haven't quite worked out where this takes place - maybe related to vega-lite/src/scale.ts?

@hydrosquall
Copy link
Member

hydrosquall commented Jan 20, 2025

Hi @dangotbanned , thanks very much for the clarifications and helpful links around expr evaluation and theme testing.

I was thinking that when altair runs in jupyter it could communicate with Vega via the Jupyter widgets API (as we did in this streamlit POC ), but since altair can run without jupyter, this isn't a global solution.

I think I see the connection you're drawing between the projects now - if there was a structured (JSON) representation of some vega JS capability, be it the colors (and sampling strategy when mixed stop counts are used) per palette, or the list of all expr, it keeps python code up to date without reimplementing each.


Stepping from the code I'm curious about the end goal. If users are able to access the color definitions in python, what would they use it for? Would they be passed directly back into an altair call, or used in other scripts?

One alternative devtool if they're not planning to use the colors right away would be to offer this info via a devtool page where you can

  • Browse all vega palette names (including bluegreen-n) varieties via selector
  • Type a palette name, get back the color codes

Alternately, we could augment the docs so clicking on a color block copies the hex code for that color to clipboard or displays in a tooltip, similar to how other color / design system sites work (e.g. material)

@hydrosquall
Copy link
Member

I hadn't noticed before writing this, but we have 300+ color scheme names and that is a huge jump from https://vega.github.io/vega/docs/schemes/

Ah good find, it looks like those color scheme names originates in vega. Upon inspection the the Vega docs list discrete (rather than interpolator function)-${n} variants of each palette behind a collapsible button, but only list the suffix rather than the full name:

https://github.com/vega/vega/blob/8fc129a6f8a11e96449c4ac0f63de0e5bfc7254c/packages/vega-typings/types/spec/scheme.d.ts#L30-L44

Image

@dangotbanned
Copy link
Member

Hi @dangotbanned , thanks very much for the clarifications and helpful links around expr evaluation and theme testing.

I was thinking that when altair runs in jupyter it could communicate with Vega via the Jupyter widgets API (as we did in this streamlit POC ), but since altair can run without jupyter, this isn't a global solution.

No worries, thanks for the detail as well @hydrosquall - interesting POC.
Reminds me of this example by @mattijn (#3630 (comment)) that uses ipywidgets (Jupyter widgets)

Gif


I think I see the connection you're drawing between the projects now - if there was a structured (JSON) representation of some vega JS capability, be it the colors (and sampling strategy when mixed stop counts are used) per palette, or the list of all expr, it keeps python code up to date without reimplementing each.

Yeah that's it 🙂


Stepping from the code I'm curious about the end goal. If users are able to access the color definitions in python, what would they use it for? Would they be passed directly back into an altair call, or used in other scripts?

Good question!
I was hoping @joelostblom had a use-case in mind when I wrote (#2617 (comment))

After some more reading - I'm finding that SchemeParams already solves the issue I had in mind.
On our side we could improve the discoverability of that via


One alternative devtool if they're not planning to use the colors right away would be to offer this info via a devtool page where you can

* Browse all vega palette names (including `bluegreen-n`) varieties via selector

* Type a palette name, get back the color codes

Alternately, we could augment the docs so clicking on a color block copies the hex code for that color to clipboard or displays in a tooltip, similar to how other color / design system sites work (e.g. material)

Big +1 from me on copy-to-clipboard for the hex color codes!

@domoritz
Copy link
Member

Can you run Vega-Lite or Vega at compile time with node and extract the colors into JSON or Python so that you don't need to do anything with javascript at runtime?

@dangotbanned
Copy link
Member

Can you run Vega-Lite or Vega at compile time with node and extract the colors into JSON or Python so that you don't need to do anything with javascript at runtime?

@domoritz that sounds like a good way forward.
I must admit I'm pretty clueless on JS, so I don't know how we'd add this into altair's CI.

I'm happy to work on how we integrate whatever the artifact is - but how to generate it is the bit I'm missing

@domoritz
Copy link
Member

The rough outline would be to create a javascript file that imports Vega and Vega-Lite, runs whatever code you need to run, and writes a JSON or Python file. Then you can run the javascript file with node.

@hydrosquall
Copy link
Member

I tried to suggest that as well, but it may have gotten lost. Thanks for raising that suggestion again.

If you can't do it at runtime, you could write a json file into Altair in a CI step.

I figure it would be something like this

// get-vega-colors.js
import vega from 'vega';
import { continuous, discrete } from `vega-scale/src/palettes`;
// write some JS to 
const myJsonToExport = {}

// get your palette names by enumerating the keys of continuous and discrete
// for all the color palette names, get the info colors from 
// vega.scheme(schemeName)

console.log(myExportJson)

Then in your CI script

node get-vega-colors.js > my-color-definitions.json

@dangotbanned
Copy link
Member

I tried to suggest that as well, but it may have gotten lost. Thanks for raising that suggestion again.

If you can't do it at runtime, you could write a json file into Altair in a CI step.

Ah apologies @hydrosquall yeah it looks like I skimmed over that part 🤦‍♂

@domoritz, @hydrosquall
How about adding something like (#2617 (comment)) to vega/vega-themes?

For vega-themes itself, I can imagine it might be more ergonomic to define a theme as transformations on scheme - rather than defining multiple variations in a script.
However, this use-case may already be covered by SchemeParams as I learned in (#2617 (comment))

@domoritz
Copy link
Member

Vega themes seems like an odd place since it's only tangentially related to schemes. I'd rather make a new package but I think doing it in Altair makes the most sense.

@jonmmease
Copy link
Contributor

vl-convert has a get_themes function for retrieving all of the available themes, do we want an analogues get_schemes method? We could dump these during build so as to not depend on vl-convert at runtime.

@dangotbanned
Copy link
Member

vl-convert has a get_themes function for retrieving all of the available themes, do we want an analogues get_schemes method? We could dump these during build so as to not depend on vl-convert at runtime.

@jonmmease that could be a good option!

Would you be able to get the contents of each scheme/palette that way?

@jonmmease
Copy link
Contributor

Would you be able to get the contents of each scheme/palette that way?

I haven't looked at this in detail, but if we can write a JavaScript snippet that builds a JavaScript object containing the names and contents of each scheme/pallete then we could retrieve that in Rust/Python.

For themes vl-convert runs this

var vegaThemes;
import('{vega_themes_url}').then((imported) => {{
    vegaThemes = imported;
}})

var themes = Object.assign({}, vegaThemes);
delete themes.version
delete themes.default

And then fetches the themes variable through JSON serialization.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement vega: vega Requires upstream action in `vega`
Projects
None yet
Development

No branches or pull requests

5 participants