v0.2.4

What's Changed

• Fix : Filename in the path by @satodayo in #1605
• fix: remove abstract class in Metric by @jjmachan in #1603
• Added pandas as a method for reading datasets by @joaorura in #1595
• fix: treat anthropic-direct API the same as bedrock in llamaindex wrapper by @ahgraber in #1584
• feat: add support for llamaindex by @jjmachan in #1609
• docs: Add embedding_model attribute when creating TestsetGenerator object by @Youngrok123 in #1606
• Fix: Add instance check before casting ChatGeneration object by @Youngrok123 in #1607
• docs: add comment for claude-3-5-sonnet-20241022 by @SokiKawashima in #1611
• docs: add community docs by @Yunnglin in #1612
• Handle embeddings for empty strings in AnswerSimilarity class by @baptiste-pasquier in #994
• Feature: batched execution by @ahgraber in #1589
• docs: how to add custom metrics by @jjmachan in #1616
• docs: added azure openai to default docs by @jjmachan in #1610
• Introduce epsilon in denominators to avoid division by zero by @jltham in #1622
• feat: Add support for llamaIndex in evaluation by @suekou in #1619
• feat: added upload for EvaluationResult by @jjmachan in #1625
• feat: automatic persona generation by @shahules786 in #1618
• docs: Refactor Code for Syntax Highlighting and URL Updates by @suekou in #1634
• fix: remove . as a split condition in noise_sensitivity.py by @jasinliu in #1627
• feat: improvements in test synthesization by @shahules786 in #1621

New Contributors

• @satodayo made their first contribution in #1605
• @joaorura made their first contribution in #1595
• @SokiKawashima made their first contribution in #1611
• @jltham made their first contribution in #1622
• @jasinliu made their first contribution in #1627

Full Changelog: v0.2.3...v0.2.4

v0.2.3

What's Changed

• feat: measure cost of testset generator by @jjmachan in #1560
• docs: added luka's blogs by @jjmachan in #1554
• Fix: add LLMContextPrecisionWithReference to init.py by @licux in #1561
• chore: add verbose option (V=1) to make targets by @trevorbowen in #1556
• add embeddings to TestsetGenerator by @hunter-walden2113 in #1562
• fixed verb agreement @ available_metrics by @gabrielhomsi in #1574
• Fix: Limit number of retries for parse failures by @rskew in #1569
• Fix: reference key error in LLMContextPrecisionWithoutReference by @xizhou-vw in #1570
• Updated return type description for evaluate function by @taihim in #1579
• Feat: add multimodal eval support by @Yunnglin in #1559
• fix: add reference_topics as default required columns in TopicAdherenceScore #1564 by @luqmansen in #1566
• fix: add reference tool call to required cols by @shahules786 in #1580
• Improve efficiency in factual correctness for precision mode by @Jeff-67 in #1578
• This commit implements the F-beta score metric by @Yuri-Albuquerque in #1543
• fix: agent goal accuracy by @shahules786 in #1583
• chores: fix pypi rendering by @shahules786 in #1581
• fix: typo: ROUGE is a metric, ROGUE is a scoundrel by @ahgraber in #1585

New Contributors

• @trevorbowen made their first contribution in #1556
• @hunter-walden2113 made their first contribution in #1562
• @gabrielhomsi made their first contribution in #1574
• @rskew made their first contribution in #1569
• @xizhou-vw made their first contribution in #1570
• @taihim made their first contribution in #1579
• @luqmansen made their first contribution in #1566
• @Yuri-Albuquerque made their first contribution in #1543

Full Changelog: v0.2.2...v0.2.3

v0.2.2

What's Changed

• docs: update summarization calculation by @jCobbSK in #1510
• fix: use get_temperature in base LLM generate by @sapphire-arches in #1520
• doc: Update modifying prompts metrics doc by @FrankyCTY in #1525
• fixbug: Expand sentence-ending punctuation check in _create_statements method by @Jeff-67 in #1523
• fix: zero division error by @shahules786 in #1530
• feat: add traces to EvaluationResult by @jjmachan in #1531
• feat: translate instruction when adapting prompt by @Yunnglin in #1529
• chore: remove old prompt object by @jjmachan in #1535
• feat: improved the testset generation to_pandas and docs by @jjmachan in #1536
• fixed typo in "single_turn_ascore" by @minorun365 in #1542
• update AspectCritic docs examples by @licux in #1547
• Fix: Resolve Agentic failure loop with improved json parsing by @ahgraber in #1541
• feat: throw error when max_token limit is reached by @jjmachan in #1549
• docs: fix Amazon bedrock info by @jjmachan in #1551
• Added community blog posts in Japanese by @minorun365 in #1550
• docs: make llms and embeddings explicit by @jjmachan in #1553

New Contributors

• @jCobbSK made their first contribution in #1510
• @sapphire-arches made their first contribution in #1520
• @FrankyCTY made their first contribution in #1525
• @Jeff-67 made their first contribution in #1523
• @Yunnglin made their first contribution in #1529
• @licux made their first contribution in #1547
• @ahgraber made their first contribution in #1541

Full Changelog: v0.2.1...v0.2.2

v0.2.1

What's Changed

• Fix Discord link by @NirantK in #1498
• Fix Quickstart link by @NirantK in #1499
• Rewrite Open Analytics by @NirantK in #1500
• Suggestions for Introduction and Key Features by @NirantK in #1501
• docs: fix broken links in testset generation custimisation by @jjmachan in #1502
• Update Bedrock modules for LangChain v0.3.x by @minorun365 in #1487
• Fixes: Quickstart Ragas Metrics by @NirantK in #1504
• Add Contributors by @NirantK in #1505
• docs: Change test_size parameter name to testset_size by @Youngrok123 in #1507
• docs: readme add llm by @shahules786 in #1511
• docs: add faq by @shahules786 in #1503
• fix: propagate callbacks in RagasOutputParser by @dlyz in #1513
• docs: fix usage of eval dataset by @shahules786 in #1514
• feat: use ensure_ascii=False for KnowledgeGraph files by @dlyz in #1518
• fix: import error for TestsetGeneration and small fixes by @jjmachan in #1516

New Contributors

• @NirantK made their first contribution in #1498
• @minorun365 made their first contribution in #1487
• @dlyz made their first contribution in #1513

Full Changelog: v0.2.0...v0.2.1

v0.2.0

What's Changed

• feat: topic adherence metric by @shahules786 in #1327
• feat: tool call accuracy by @shahules786 in #1311
• feat: sql sementic equivalence by @shahules786 in #1323
• feat: Factual correctnes metric by @shahules786 in #1334
• docs: migrating over to mkdocs by @jjmachan in #1301
• feat: Adding transforms for testset generation by @jjmachan in #1344
• docs: traditional metrics by @shahules786 in #1346
• feat: transforms for Knowledge Graphs by @jjmachan in #1345
• docs: core concepts in metrics by @shahules786 in #1347
• chores: cleanup metrics by @shahules786 in #1348
• docs: refactor core concepts by @shahules786 in #1351
• wrap as_complete with async by @liupgd in #1350
• chores: add datasets to init by @shahules786 in #1356
• docs: new getting started by @shahules786 in #1355
• docs: metrics concepts by @shahules786 in #1361
• docs: add mindmap by @shahules786 in #1362
• Fixed Nav, search, and some styling issues for the docs by @joelk9895 in #1363
• feat: transform engine to execute transforms by @jjmachan in #1352
• chores: remove fuzzywuzzy by @shahules786 in #1369
• docs: component guides by @shahules786 in #1365
• chores: fix typo by @shahules786 in #1370
• fix:ensure correct sequence in tool calls by @shahules786 in #1371
• [howtos] update Langfuse cookbook by @jannikmaierhoefer in #1368
• feat: adding an implementation of abstractQA by @jjmachan in #1359
• feat: implement TestsetGenerator by @jjmachan in #1375
• docs: testset generation by @shahules786 in #1373
• feat: added more simulators by @jjmachan in #1376
• Update models in Ragas documentation: change gpt-3.5-turbo-16k to gpt-4o-mini and gpt-4 to gpt-4o by @suekou in #1380
• fix: aspect_critique documentation by @jjmachan in #1382
• docs: added google analytics to mkdocs by @jjmachan in #1384
• docs: fix index for metric/index.md by @jjmachan in #1388
• fix: fixed verdict comparison in evolutions.py by @MarkHmnv in #1366
• feat: output new Testset object for generation by @jjmachan in #1386
• Update answer_relevance.md: Minor spell correct by @soumendrak in #1390
• feat: support for callbacks and traces in testset generation by @jjmachan in #1392
• non-asii characters support in faithfulness by @gargewp in #1396
• feat: get and set prompts mixin by @shahules786 in #1391
• fixes: rename naming in testset generation by @shahules786 in #1393
• chore!: moving to pydantic2 by @jjmachan in #1394
• docs: improve getting started by @shahules786 in #1398
• Fix typo in metric error message by @suekou in #1404
• feat!: moving experimental.testset to replace old testset by @jjmachan in #1401
• feat: renamed sample operations and cleaned up test warnings by @jjmachan in #1407
• added better styling to reder jupyter notebooks nicely. by @zodwick in #1389
• docs: add core concepts for test generation by @shahules786 in #1402
• docs:Auto Gen of Reference Page by @zodwick in #1408
• fix key error for FaithfulnesswithHHEM by @linzzzzzz in #1410
• fix: remove usage of contexts in response relevancy by @shahules786 in #1409
• fix: score for jupyter by @shahules786 in #1411
• feat: NER extractor and Jaccard by @shahules786 in #1406
• feat(styles): refactor CSS to use custom variables by @zodwick in #1413
• docs : add customisation by @shahules786 in #1412
• feat(docs): add support for reusable snippets by @zodwick in #1414
• Feature/css variables by @zodwick in #1415
• docs: fix nav bar by @shahules786 in #1416
• feat: more features to prompt object by @jjmachan in #1418
• chore: add tracking for new testset generation by @jjmachan in #1419
• feat: adapt prompt to different languages by @jjmachan in #1429
• chores: add reference to gitignore by @shahules786 in #1428
• docs: redo intro by @shahules786 in #1427
• Add Custom Ragas Light and Dark Themes (v1) by @zodwick in #1431
• fix attribute error for faithfulness metric by @linzzzzzz in #1436
• docs: language adaptation by @shahules786 in #1446
• fix: data export by @shahules786 in #1461
• feat: save and load Prompts by @jjmachan in #1458
• refactor: convert faithfulness, answer_correctness and noise_sensitivity to new prompt object by @jjmachan in #1462
• Minor typo in _answer_relevance.py by @pjanowski in #1452
• feat: added repr for prompt objects by @jjmachan in #1459
• feat: change context recall and precision prompts to new by @jjmachan in #1467
• docs: fix invalid links and imports by @shahules786 in #1473
• feat: changed summary to new prompt by @jjmachan in #1469
• docs: redo readme by @shahules786 in #1480
• docs: new build process with rtd and API documentation by @jjmachan in #1474
• chore: added install for mkdocs insiders by @jjmachan in #1481
• chores: update gitignore by @shahules786 in #1483
• docs: improve testgen namings by @shahules786 in #1489
• Fixed an issue where non-ASCII characters were changed to Unicode characters within the prompt by @Youngrok123 in #1490
• docs: fix search by @shahules786 in #1491
• feat: added save and load to RagasDataset by @jjmachan in #1492
• docs: moved from ipnb files to md files by @jjmachan in #1482
• feat: set ensure_ascii to false by @jjmachan in #1494
• feat: small refactors and cleanups by @jjmachan in #1493

New Contributors

• @liupgd made their first contribution in #1350
• @joelk9895 made their first contribution in #1363
• @jannikmaierhoefer made their first contribution in #1368
• @suekou made their first contribution in #1380
• @MarkHmnv made their first contribution in #1366
• @soumendrak made their first contribution in #1390
• @gargewp made their first contribution in #1396
• @zodwick made their first contribution in #1389
• @linzzzzzz made their first contribution in https://git...

v0.2.0-beta

What's Changed

• feat: topic adherence metric by @shahules786 in #1327
• feat: tool call accuracy by @shahules786 in #1311
• feat: sql sementic equivalence by @shahules786 in #1323
• feat: Factual correctnes metric by @shahules786 in #1334
• docs: migrating over to mkdocs by @jjmachan in #1301
• feat: Adding transforms for testset generation by @jjmachan in #1344
• docs: traditional metrics by @shahules786 in #1346
• feat: transforms for Knowledge Graphs by @jjmachan in #1345
• docs: core concepts in metrics by @shahules786 in #1347
• chores: cleanup metrics by @shahules786 in #1348
• docs: refactor core concepts by @shahules786 in #1351
• wrap as_complete with async by @liupgd in #1350
• chores: add datasets to init by @shahules786 in #1356
• docs: new getting started by @shahules786 in #1355
• docs: metrics concepts by @shahules786 in #1361
• docs: add mindmap by @shahules786 in #1362
• Fixed Nav, search, and some styling issues for the docs by @joelk9895 in #1363
• feat: transform engine to execute transforms by @jjmachan in #1352
• chores: remove fuzzywuzzy by @shahules786 in #1369
• docs: component guides by @shahules786 in #1365
• chores: fix typo by @shahules786 in #1370
• fix:ensure correct sequence in tool calls by @shahules786 in #1371
• [howtos] update Langfuse cookbook by @jannikmaierhoefer in #1368
• feat: adding an implementation of abstractQA by @jjmachan in #1359
• feat: implement TestsetGenerator by @jjmachan in #1375
• docs: testset generation by @shahules786 in #1373
• feat: added more simulators by @jjmachan in #1376
• Update models in Ragas documentation: change gpt-3.5-turbo-16k to gpt-4o-mini and gpt-4 to gpt-4o by @suekou in #1380
• fix: aspect_critique documentation by @jjmachan in #1382
• docs: added google analytics to mkdocs by @jjmachan in #1384
• docs: fix index for metric/index.md by @jjmachan in #1388
• fix: fixed verdict comparison in evolutions.py by @MarkHmnv in #1366
• feat: output new Testset object for generation by @jjmachan in #1386
• Update answer_relevance.md: Minor spell correct by @soumendrak in #1390
• feat: support for callbacks and traces in testset generation by @jjmachan in #1392
• non-asii characters support in faithfulness by @gargewp in #1396
• feat: get and set prompts mixin by @shahules786 in #1391
• fixes: rename naming in testset generation by @shahules786 in #1393
• chore!: moving to pydantic2 by @jjmachan in #1394
• docs: improve getting started by @shahules786 in #1398
• Fix typo in metric error message by @suekou in #1404
• feat!: moving experimental.testset to replace old testset by @jjmachan in #1401
• feat: renamed sample operations and cleaned up test warnings by @jjmachan in #1407
• added better styling to reder jupyter notebooks nicely. by @zodwick in #1389
• docs: add core concepts for test generation by @shahules786 in #1402
• docs:Auto Gen of Reference Page by @zodwick in #1408
• fix key error for FaithfulnesswithHHEM by @linzzzzzz in #1410
• fix: remove usage of contexts in response relevancy by @shahules786 in #1409
• fix: score for jupyter by @shahules786 in #1411
• feat: NER extractor and Jaccard by @shahules786 in #1406
• feat(styles): refactor CSS to use custom variables by @zodwick in #1413
• docs : add customisation by @shahules786 in #1412
• feat(docs): add support for reusable snippets by @zodwick in #1414
• Feature/css variables by @zodwick in #1415
• docs: fix nav bar by @shahules786 in #1416
• feat: more features to prompt object by @jjmachan in #1418
• chore: add tracking for new testset generation by @jjmachan in #1419

New Contributors

• @liupgd made their first contribution in #1350
• @joelk9895 made their first contribution in #1363
• @jannikmaierhoefer made their first contribution in #1368
• @suekou made their first contribution in #1380
• @MarkHmnv made their first contribution in #1366
• @soumendrak made their first contribution in #1390
• @gargewp made their first contribution in #1396
• @zodwick made their first contribution in #1389
• @linzzzzzz made their first contribution in #1410

Full Changelog: v0.1.20...v0.2.0-beta

v0.1.21

Full Changelog: v0.1.16...v0.1.21

• Pins to langchain<0.3
• based on v0.1.16 which was the last working version for 0.1
• hopefully last release before v0.2

v0.1.20

What's Changed

• fix: pin langchain_core to <3 by @jjmachan in #1329

Full Changelog: v0.1.19...v0.1.20

v0.1.19

What's Changed

• feat: add non llm based context recall by @shahules786 in #1266
• feat: Non LLM based context precision by @shahules786 in #1264
• feat: added n and temperature passthrough to prompt by @jjmachan in #1281
• Add option to disable progress bar in evaluate by @freddiev4 in #1279
• feat: new aspect critic metrics by @shahules786 in #1286
• feat: add simple scorer metric by @shahules786 in #1291
• docs: add docs for new metrics by @shahules786 in #1292
• Fix wrong input key name by @ruanhao in #1300
• chore: move rubrics based metric into single file by @shahules786 in #1287
• feat: Instance specifc rubrics metrics by @shahules786 in #1304
• feat: agent goal accuracy metric by @shahules786 in #1303
• docs: agent metrics by @shahules786 in #1314
• Update Opik notebook example and integration by @jverre in #1295
• fix: column rename by @shahules786 in #1324

New Contributors

• @freddiev4 made their first contribution in #1279
• @ruanhao made their first contribution in #1300

Full Changelog: v0.1.18...v0.1.19

v0.1.18

What's Changed

• fix: inverse distance by @shahules786 in #1267
• fix: non llm based metrics by @shahules786 in #1268
• fix: v1 to v2 dataset by @shahules786 in #1275
• fix: make score nested if loop_is_running by @jjmachan in #1276

Full Changelog: v0.1.17...v0.1.18