Börje F. Karlsson (申博野)

Principal Member of Technical Staff
Beijing Academy of Artificial Intelligence (BAAI)

email:	borje{dot}karlsson{at}tellarin.com
website:	Börje F. Karlsson
address:	Beijing Academy of Artificial Intelligence 150 Chengfu Road, 5F Haidian District Beijing, P.R. China 100080
links:

Short Biography

I am currently Principal Member of Technical Staff at the Beijing Academy of Artificial Intelligence (BAAI), a non-profit research institute in Beijing, China, where I lead a team working on new AI scenarios involving intelligent agents, embodied AI, and multi-modality models. My main interests include machine learning systems, multi-modality reasoning, intelligent agents (both physical and digital), multilingual knowledge pipelines, and digital entertainment.

From 2012 to 2023, I was Principal Research Engineer Lead at Microsoft Research Asia (MSRA), leading multiple research projects and technology transfer initiatives in partnership with different product teams (Windows, Azure, Bing, Excel, etc.). Initially in the Mobile and Sensing Systems and System Algorithms groups, then in the Knowledge Computing group – part of Data, Knowledge, and Intelligence (DKI), my research has been published in top conferences and journals in systems and AI, and our generated technologies have been open-sourced or integrated into diverse Microsoft products and services, and used by millions of users worldwide.

Before MSRA I was a Research Scientist at Nokia Institute of Technology (INdT) - a Nokia-founded R&D lab in Brazil, where I estabilished and led its Contextual Intelligence Platform group. Previously I’ve also worked as researcher, system/software architect, consultant, product owner, and development team lead on projects for companies such as Nokia, C.E.S.A.R., Motorola, 3, TIM, Radix.com, Petrobras, among others.

I've been a member of the organizing committee and TPC of conferences, including ICLR, AAAI, ACL, IJCAI, MobileSoft, SIGGRAPH, SIGGRAPH Asia's MGIA, IFIP ICEC, etc. I received my doctorate in Computer Science from PUC-Rio, Brazil and my BSc from UFPE, also in Brazil.

Research » students

My research interests lie broadly in machine learning systems, multi-modality knowledge mining, mobile/pervasive computing, and digital entertainment, with a recent focus on agentic systems, embodied egocentric AI, and multi-cultural multimodal models.

Some previous at MSRA projects include: tabular data semantics, design intelligence, modular KBQA, real-world information extraction, and location-based systems (e.g., navigation, mapping, etc.).

News

Aug-2026: EgoActor and the new version of SWITCH have just been accepted at EMNLP 2026.
May-2026: Our colab paper, ICPRL, has just been accepted for presentation at the LM4Plan Workshop at ICML 2026.
Apr-2026: X-DiffVLA has been accepted at RSS 2026.
Mar-2026: The SWITCH (v0) benchmark has been accepted for presentation at the LLA Workshop at ICLR 2026.
Jan-2026: RANGER and Proprio-MLLM have both been accepted for presentation at ICRA 2026.
Nov-2025: Our colab benchmark paper, DeepPHY, has just been accepted for presentation at AAAI 2026.
Aug-2025: MMDiag has been accepted for presentation at EMNLP 2025.
Jun-2025: DualTHOR is now out on GitHub. Pre-print on arXiv.
May-2025: Cradle has been accepted for presentation at ICML 2025.
And SEA-VL has been accepted for presentation at ACL 2025.
Apr-2025: Our collaborative work Aya Dataset is listed as one of the most notable AI releases of 2024 in Stanford University's AI Index 2025 Annual Report” (1 of 2 datasets selected, and 1 of 32 total releases listed).
Jan-2025: MART and INCLUDE (spotlight) have both been accepted for presentation at ICLR 2025.

HIRING:
Internship - My group is looking for self-motivated student candidates for multiple research or engineering internship spots: Internship opportunities in our lab (PDF).
FTE - We're also hiring strong researchers and engineers as full time employees at BAAI. Please shoot me an e-mail!

» full list Selected publications

[arXiv] LectūraAgents: A Multi-Agent Framework for Adaptive Personalized AI-Assisted Learning and Embodied Teaching [paper] [bib]

Jaward Sesay, Yue Yu, Siwei Dong, Börje F. Karlsson.
arXiv:2606.16428, June 2026

[arXiv] ExoActor: Exocentric Video Generation as Generalizable Interactive Humanoid Control [paper] [bib]

Yanghao Zhou, Jingyu Ma, Yibo Peng, Zhenguo Sun, Yu Bai, Börje F. Karlsson.
arXiv:2604.27711, April 2026

[ICRA'26] RANGER: A Monocular Zero-Shot Semantic Navigation Framework through Contextual Adaptation [paper] [bib]

Mingming Yu, Yi Chen, Börje F. Karlsson, Wenjun Wu.
IEEE ICRA , 2026

[AAAI'26] DeepPHY: Benchmarking Agentic VLMs on Physical Reasoning [paper] [bib]

Xinrun Xu, Pi Bu, Ye Wang, Börje F. Karlsson, Ziming Wang, Tengtao Song, Qi Zhu, Jun Song, Zhiming Ding, Bo Zheng.
AAAI Conference on Artificial Intelligence, 2026

[EMNLP'26] SWITCH: Benchmarking Modeling and Handling of Tangible Interfaces in Long-horizon Embodied Scenarios [paper] [bib]

Juntao Cheng, Wanyue Zhang, Zhiwei Yu, Shuo Ren, Zheqi He, Shaoxuan Xie, Guocai Yao, Jieru Lin, Börje F. Karlsson, Jiajun Zhang.
EMNLP'26 - Findings, 2026

[arXiv] Being-0: A Humanoid Robotic Agent with Vision-Language Models and Modular Skills [paper] [bib]

Haoqi Yuan, Yu Bai, Yuhui Fu, Bohan Zhou, Yicheng Feng, Xinrun Xu, Yi Zhan, Börje F. Karlsson, Zongqing Lu.
arXiv:2503.12533, March 2025

[ACL'25] Crowdsource, Crawl, or Generate? Creating SEA-VL, a Multicultural Vision-Language Dataset for Southeast Asia [paper] [bib]

Samuel Cahyawijaya, Holy Lovenia, Joel Ruben Antony Moniz, Tack Hwa Wong, Mohammad Rifqi Farhansyah, Thant Thiri Maung, Frederikus Hudi, David Anugraha, Muhammad Ravi Shulthan Habibi, Muhammad Reza Qorib, Amit Agarwal, Joseph Marvin Imperial, Hitesh Laxmichand Patel, Vicky Feliren, Bahrul Ilmi Nasution, Manuel Antonio Rufino, Genta Indra Winata, Rian Adam Rajagede, Carlos Rafael Catalan, Mohamed Fazli Imam, Priyaranjan Pattnayak, Salsabila Zahirah Pranida, Kevin Pratama, Yeshil Bangera, Adisai Na-Thalang, Patricia Nicole Monderin, Yueqi Song, Christian Simon, Lynnette Hui Xian Ng, Richardy Lobo' Sapan, Taki Hasan Rafi, Bin Wang, Supryadi, Kanyakorn Veerakanjana, Piyalitt Ittichaiwong, Matthew Theodore Roque, Karissa Vincentio, Takdanai Kreangphet, Phakphum Artkaew, Kadek Hendrawan Palgunadi, Yanzhi Yu, Rochana Prih Hastuti, William Nixon, Mithil Bangera, Adrian Xuan Wei Lim, Aye Hninn Khine, Hanif Muhammad Zhafran, Teddy Ferdinan, Audra Aurora Izzani, Ayushman Singh, Evan, Jauza Akbar Krito, Michael Anugraha, Fenal Ashokbhai Ilasariya, Haochen Li, John Amadeo Daniswara, Filbert Aurelian Tjiaranata, Eryawan Presma Yulianrifat, Can Udomcharoenchaikit, Fadil Risdian Ansori, Mahardika Krisna Ihsani, Giang Nguyen, Anab Maulana Barik, Dan John Velasco, Rifo Ahmad Genadi, Saptarshi Saha, Chengwei Wei, Isaiah Flores, Kenneth Ko Han Chen, Anjela Gail Santos, Wan Shen Lim, Kaung Si Phyo, Tim Santos, Meisyarah Dwiastuti, Jiayun Luo, Jan Christian Blaise Cruz, Ming Shan Hee, Ikhlasul Akmal Hanif, M.Alif Al Hakim, Muhammad Rizky Sya'ban, Kun Kerdthaisong, Lester James V. Miranda, Fajri Koto, Tirana Noor Fatyanosa, Alham Fikri Aji, Jostin Jerico Rosal, Jun Kevin, Robert Wijaya, Onno P. Kampman, Ruochen Zhang, Börje F. Karlsson, Peerat Limkonchotiwat.
Annual Meeting of the Association for Computational Linguistics, 2025

[ICLR'25] MLLM as Retriever: Interactively Learning Multimodal Retrieval for Embodied Agents [paper] [bib]

Junpeng Yue, Xinrun Xu, Börje F. Karlsson, Zongqing Lu.
International Conference on Learning Representations, 2025

[ICLR'25] INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge [paper] [bib]

Angelika Romanou, Negar Foroutan, Anna Sotnikova, Sree Harsha Nelaturu, Shivalika Singh, Rishabh Maheshwary, Micol Altomare, Zeming Chen, Mohamed A. Haggag, Snegha A, Alfonso Amayuelas, Azril Hafizi Amirudin, Danylo Boiko, Michael Chang, Jenny Chim, Gal Cohen, Aditya Kumar Dalmia, Abraham Diress, Sharad Duwal, Daniil Dzenhaliou, Daniel Fernando Erazo Florez, Fabian Farestam, Joseph Marvin Imperial, Shayekh Bin Islam, Perttu Isotalo, Maral Jabbarishiviari, Börje F. Karlsson, Eldar Khalilov, Christopher Klamm, Fajri Koto, Dominik Krzemiński, Gabriel Adriano de Melo, Syrielle Montariol, Yiyang Nan, Joel Niklaus, Jekaterina Novikova, Johan Samir Obando Ceron, Debjit Paul, Esther Ploeger, Jebish Purbey, Swati Rajwal, Selvan Sunitha Ravi, Sara Rydell, Roshan Santhosh, Drishti Sharma, Marjana Prifti Skenduli, Arshia Soltani Moakhar, Bardia Soltani Moakhar, Ayush Kumar Tarun, Azmine Toushik Wasi, Thenuka Ovin Weerasinghe, Serhan Yilmaz, Mike Zhang, Imanol Schlag, Marzieh Fadaee, Sara Hooker, Antoine Bosselut.
International Conference on Learning Representations, 2025

[ICML'25] Cradle: Empowering Foundation Agents Towards General Computer Control [paper] [bib]

Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia, Ziluo Ding, Boyu Li, Bohan Zhou, Junpeng Yue, Jiechuan Jiang, Yewen Li, Ruyi An, Molei Qin, Chuqiao Zong, Longtao Zheng, Yujie Wu, Xiaoqiang Chai, Yifei Bi, Tianbao Xie, Pengjie Gu, Xiyun Li, Ceyao Zhang, Long Tian, Chaojie Wang, Xinrun Wang, Börje F. Karlsson, Bo An, Shuicheng Yan, Zongqing Lu.
International Conference on Machine Learning, 2025

[EMNLP'24] SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages [paper] [bib]

Holy Lovenia, Rahmad Mahendra, Salsabil Maulana Akbar, Lester James V. Miranda, Jennifer Santoso, Elyanah Aco, Akhdan Fadhilah, Jonibek Mansurov, Joseph Marvin Imperial, Onno P. Kampman, Joel Ruben Antony Moniz, Muhammad Ravi Shulthan Habibi, Frederikus Hudi, Railey Montalan, Ryan Ignatius, Joanito Agili Lopo, William Nixon, Börje F. Karlsson, James Jaya, Ryandito Diandaru, Yuze Gao, Patrick Amadeus, Bin Wang, Jan Christian Blaise Cruz, Chenxi Whitehouse, Ivan Halim Parmonangan, Maria Khelli, Wenyu Zhang, Lucky Susanto, Reynard Adha Ryanda, Sonny Lazuardi Hermawan, Dan John Velasco, Muhammad Dehan Al Kautsar, Willy Fitra Hendria, Yasmin Moslem, Noah Flynn, Muhammad Farid Adilazuarda, Haochen Li, Johanes Lee, R. Damanhuri, Shuo Sun, Muhammad Reza Qorib, Amirbek Djanibekov, Wei Qi Leong, Quyet V. Do, Niklas Muennighoff, Tanrada Pansuwan, Ilham Firdausi Putra, Yan Xu, Ngee Chia Tai, Ayu Purwarianti, Sebastian Ruder, William Tjhi, Peerat Limkonchotiwat, Alham Fikri Aji, Sedrick Keh, Genta Indra Winata, Ruochen Zhang, Fajri Koto, Zheng-Xin Yong, Samuel Cahyawijaya.
Conference on Empirical Methods in Natural Language Processing, 2024

[LLM Agents @ ICLR'24, BAAI Tech Report] Towards General Computer Control: A Multimodal Agent for Red Dead Redemption II as a Case Study [paper] [bib]

Weihao Tan, Ziluo Ding, Wentao Zhang, Boyu Li, Bohan Zhou, Junpeng Yue, Haochong Xia, Jiechuan Jiang, Longtao Zheng, Xinrun Xu, Yifei Bi, Pengjie Gu, Xinrun Wang, Börje F. Karlsson, Bo An, Zongqing Lu.
ICLR 2024 Workshop on LLM Agents, May 2024
Beijing Academy of Artificial Intelligence Technical Report, February 2024

[TASLP] Decomposed Meta-Learning for Few-Shot Sequence Labeling [paper] [bib]

Tingting Ma, Qianhui Wu, Huiqiang Jiang, Jieru Lin, Börje F. Karlsson, Tiejun Zhao, Chin-Yew Lin.
IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP), February 2024

[ACL'24] Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning [paper] [bib]

Shivalika Singh*, Freddie Vargus*, Daniel D’souza*, Börje F. Karlsson*, Abinaya Mahendiran*, Wei-Yin Ko*, Herumb Shandilya*, Jay Patel, Deividas Mataciunas, Laura OMahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Souza Moura, Dominik Krzemiński, Hakimeh Fadaei, Irem Ergün, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Vu Minh Chien, Sebastian Ruder, Surya Guthikonda, Emad A Alghamdi, Sebastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzieh Fadaee, Sara Hooker.
Annual Meeting of the Association for Computational Linguistics , 2024

[BAAI Tech Report, arXiv] A Survey on Game Playing Agents and Large Models: Methods, Applications, and Challenges [paper] [bib]

Xinrun Xu, Yuxin Wang, Chaoyi Xu, Ziluo Ding, Jiechuan Jiang, Zhiming Ding, Börje F. Karlsson.
Beijing Academy of Artificial Intelligence Technical Report, February 2024
arXiv:2403.10249, February 2024

[NAACL'24] Universal NER: A Gold-Standard Multilingual Named Entity Recognition Benchmark [paper] [bib]

Stephen Mayhew, Terra Blevins, Shuheng Liu, Marek Šuppa, Hila Gonen, Joseph M. Imperial, Börje F. Karlsson, Peiqin Lin, Nikola Ljubešic, LJ Miranda, Barbara Plank, Arij Riabi, Yuval Pinter.
Conference of the North American Chapter of the Association for Computational Linguistics, 2024

[IJCAI'24] AutoAgents: A Framework for Automatic Agent Generation [paper] [bib]

Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, Börje F. Karlsson, Jie Fu, Yemin Shi.
International Joint Conference on Artificial Intelligence, 2024

[ACL'23] CoLaDa: A Collaborative Label Denoising Framework for Cross-lingual Named Entity Recognition [paper] [bib]

Tingting Ma, Qianhui Wu, Huiqiang Jiang, Börje F. Karlsson, Tiejun Zhao, Chin-Yew Lin.
Annual Meeting of the Association for Computational Linguistics, 2023

[MSR Tech Report] Dataset and Baseline System for Multi-lingual Extraction and Normalization of Temporal and Numerical Expressions [paper] [bib]

Sanxing Chen, Yongqiang Chen, Börje F. Karlsson.
Microsof Research Technical Report MSR-TR-2023-9, February 2023

[EMNLP'22] TIARA: Multi-grained Retrieval for Robust Question Answering over Large Knowledge Bases [paper] [bib]

Yiheng Shu, Zhiwei Yu, Yuhan Li, Börje F. Karlsson, Tingting Ma, Yuzhong Qu, Chin-Yew Lin.
Conference on Empirical Methods in Natural Language Processing, 2022

[ACL'21] AdvPicker: Effectively Leveraging Unlabeled Data via Adversarial Discriminator for Cross-Lingual NER [paper] [bib]

Weile Chen, Huiqiang Jiang, Qianhui Wu, Börje F. Karlsson, Yi Guan.
Annual Meeting of the Association for Computational Linguistics, 2021

[IJCAI'20] UniTrans: Unifying Model Transfer and Data Transfer for Cross-Lingual Named Entity Recognition with Unlabeled Data [paper] [bib]

Qianhui Wu, Zijia Lin, Börje F. Karlsson, Biqing Huang, Jian-Guang Lou.
International Joint Conferences on Artifical Intelligence, 2020

[ACL'20] Single-/Multi-Source Cross-Lingual NER via Teacher-Student Learning on Unlabeled Data in Target Language [paper] [bib]

Qianhui Wu, Zijia Lin, Börje F. Karlsson, Jian-Guang Lou, Biqing Huang.
Annual Meeting of the Association for Computational Linguistics, 2020

[AAAI'19] GRN: Gated Relation Network to Enhance Convolutional Neural Network for Named Entity Recognition [paper] [bib]

Hui Chen, Zijia Lin, Guiguang Ding, Jian-Guang Lou, Yusen Zhang, Börje F. Karlsson.
AAAI Conference on Artificial Intelligence, 2019

[NAACL'19] CAN-NER: Convolutional Attention Network for Chinese Named Entity Recognition [paper] [bib]

Yuying Zhu, Guoxin Wang, Börje F. Karlsson.
Conference of the North American Chapter of the Association for Computational Linguistics, 2019

[INFOCOM'19] Incrementally-deployable Indoor Navigation with Automatic Trace Generation [paper] [bib]

Yuanchao Shu, Zhuqi Li, Börje F. Karlsson, Yiyong Lin, Thomas Moscibroda, Kang Shin.
IEEE International Conference on Computer Communications, 2019

[MobiCom'17] Demo: Towards Flexible and Scalable Indoor Navigation [paper] [bib]

Zhuqi Li, Yuanchao Shu, Börje F Karlsson, Yiyong Lin, Thomas Moscibroda.
ACM International Conference on Mobile Computing and Networking, 2017

[Microsoft Research Blog] Path Guide: A New Approach to Indoor Navigation [webpage]

Yuanchao Shu, and Börje F. Karlsson.
Microsoft Research Blog, July 14, 2017

[BuildSys'16] Systematically Debugging IoT Control System Correctness for Building Automation [paper] [bib]

Chieh-Jan Mike Liang, Lei Bu, Zhao Li, Junbei Zhang, Shi Han, Börje F. Karlsson, Dongmei Zhang, Feng Zhao.
ACM International Conference on Systems for Energy-Efficient Built Environments, 2016

[IPSN'15] SIFT: Building an Internet of Safe Things [paper] [bib]

Chieh-Jan Mike Liang, Börje F. Karlsson, Nicholas D Lane, Feng Zhao, Junbei Zhang, Zheyi Pan, Zhao Li, Yong Yu.
ACM International Conference on Information Processing in Sensor Networks, 2015

[MobiCom'14] Caiipa: Automated Large-scale Mobile App Testing through Contextual Fuzzing [paper] [bib]

Chieh-Jan Mike Liang, Nicholas D Lane, Niels Brouwers, Li Zhang, Börje F. Karlsson, Hao Liu, Yan Liu, Jun Tang, Xiang Shan, Ranveer Chandra, Feng Zhao.
ACM International Conference on Mobile Computing and Networking, 2014

Academic Service (recent)

Organizing Committees

ACM SIGGRAPH Asia’s Symposium on Mobile Graphics and Interactive Applications (MGIA) – Co-chair 2016, Chair 2014, Chair 2013

SBC SBGames 2013 – Tutorials Co-chair

ACM SIGGRAPH 2012 – Operations Sub-committee

Technical Program Committees

2025: IJCAI PC, ACL RR (Area Chair), NAACL, ACL, EMNLP, EACL, IFIP ICEC

2024: IJCAI PC, ACL RR, NAACL, ACL, EMNLP, IFIP ICEC, AfricaNLP Workshop @ ICLR

2023: ACL, EACL, AACL, ACL RR

2022: AAAI, IJCAI PC Board, ACL, NAACL, EMNLP, AACL, ACL RR

2021: AAAI, IJCAI Senior PC, ACL

2020: ACL, AACL-IJCNLP

2018: IEEE/ACM MobileSoft, IFIP ICEC, SIGGRAPH Asia – Emerging Technologies

2017: IEEE/ACM MobileSoft, ACM SIGGRAPH Asia’s MGIA, ACM Multimedia – Systems track

2016: IEEE CPSCom

2015: IEEE/ACM MobileSoft, IFIP ICEC, ACM SIGGRAPH Asia’s MGIA

2014: IEEE/ACM MobileSoft, IFIP ICEC

2013: IFIP ICEC

Guest co-editor — Computer & Graphics, Special Section on Recent Mobile Graphics Research, 2016/2017

Reviewer

Journals:
Neurocomputing (2021-2025), Journal of Open Source Software (2023-2025), Entertainment Computing (2025, 2023, 2016, 2015), ACM/IEEE TASLP (2023-2024), IEEE Transactions on Mobile Computing (2014-2018), ACM IMWUT – Interactive, Mobile, Wearable and Ubiquitous Technologies (2017), SBC RITA (2015, 2006-2011)

Conferences and Workshops:
ICLR (2024-2026), COLM (2025), ICCV (2025), DL4C Workshop @ NeurIPS 2025, Tiny Papers @ ICLR (2024), Generative Models for Decision Making Workshop - GenAI4DM @ ICLR (2024),
AAAI (2019, 2017, 2014, 2010), IJCAI (2018, 2013, 2009), IEEE WoWMoM (2015), Mobiquitous (2014), MobiCASE (2013)

In my free time, I enjoy travelling and trying different foods and beer.

Site design originated from Alex Halderman.
Last modified: Mar. 2025