In CourseMill 7.1: FAIL + FAIL = PASS

Wow this one sounds like an interesting one! I would like to look at the SCORM package but it almost sounds like the "lowest passing score" in the SCORM is being met. What where the scores you received each time?

undefined

Can you submit your SCORM package at http://cds.trivantis.com/ and I can take a look?

Or if you want to post it here, either is fine with me.

OK, did some testing and some interesting results

Test 1

Failed score=0

Retried test

Failed score=0

Clicked finished to close course

CourseMill status=incomplete score=0

Relaunched course

Failed score=0

Clicked finished to close course

CourseMill status=incomplete

Test 2

Failed score=40

Retried test

Failed score=40

Clicked finished to close course

CourseMill status=complete score=100

With the first test getting no completion and score in CourseMill and the second test getting points that could total to a passing course when you add both attempts, it makes me think some manual completion logic is in this course. If this where a Lectora course I would be looking for actions that add up the "points" and possibly setting the score to 100.

However, the logs that are produced by Articulate don't show the score being set to 100% and that happening. I wonder if there is and logic on the finish button to set score to 100% when some conditions are met.

Side note: it's very strange that Articulate is encrypting the suspend data.

Thanks for offering to take a look. I'm attaching the SCORM package. This version is set up in debug mode, so a second window containing the log will open when you run it. (The folks at Articulate gave me a tip on how to do that.)

I did find a bunch of "intSCORMError=401" errors in the log, and I'm trying to figure that out -- but frankly, that's not my strong suit. I found in the Rustici "Run Time Reference this definition, though:

A few other notes ...

The quiz is set to report "Incomplete/Completed"

The issue seems to show up only when retrying the quiz, or, if you exit the quiz and come back, if you Resume.

As for scores, I purposely fail the first attempt at 20% (1 of 5). On the retry it doesn't seem to matter whether you answer none or less than 4 correctly. The score is always shown as 100 in CourseMill. And one other note; even though it's set to report as Incomplete/Completed, the status shows up in LMS as Passed.

Something is defiantly strange with the scoring in thing SCORM package. I put this in the SCORM cloud and had the below test

Test 3 SCORM cloud

Take test for a 20% score

Retake test for a 40% score

Close the course and SCORM cloud had my score at 60%

So it looks like the scores from each attempt are a cumulative for the time that the test is being taken.

Note SCORM cloud does work with the mastery score and CourseMill does.

Good morning ...

I replicated your 2 tests with these results:

Test 1

Fail w/ score of 0, clicked finished:

%Completed =10%

Status = Started

Score = 0

Reentered quiz, filed w/ score of 0, clicked finish:

%Completed =10%

Status = Started

Score = 0

Test 2

Failed w/ score of 40

Retried Quiz

Failed w/ score of 40

Clicked Finish

%Completed =100%

Status = Passed

Score = 100

So the only difference is that I'm seeing "started" vs "incomplete" and "passed" instead of "completed." I wonder whether we're on the same version of CourseMill? When I log onto CourseMill, the browser tab briefly shows "7.1." If you're on a different build, I wonder if that makes a difference.

To your comment about cumulative scores -- when this issue first came up, I thought the same thing, and so did the host of the LMS. But that didn't pan out; by failing twice with a score of 10% each time, that'd still be a failing score of 20%, yet the LMS records 100%.

That's an interesting thought on the Finish button. But if memory serves, earlier I tested by simply closing the browser instead of the finish button, and I recall the results were the same.

Now, when I ran the debug log that I attached in the Word document, I tried to be clever in being able to figure things out.

Since each line is time-stamped, I waited for a new minute to come around before taking the next action. I figured that by doing that, I could isolate each action to a section of the log.

From my notes:

1424: entered quiz

1425: answer Q1 C

1426: answer Q2 I

1427: answer Q3 I

1428: answer Q4 I

1429: answer Q5 I

1430: clicked Retry Quiz

1431: answer Q1 I

1432: answer Q2 I

1433: answer Q3 I

1434: answer Q4 C

1435: answer Q5 C

1436: clicked Finish

So, all the activity on the log related to the Finish button would be time-stamped at 1436.

I hope this helps -- and thanks again for looking.

Dave

undefined

SCORM Cloud does not support the mastery score.

undefined

Yeah?? looking at the transcript for my attempts and your, and it is always showing three extra scores, a 50, a 66 and a 100%. It looks like these score are sent a few times from the content. Screenshot is from the CourseMill Full Debug log.

undefined

Has this been brought up with Articulate support yet?

I also notice in SCORM cloud that the Cloud debugger is seeing additional scores beyond the score that the course is showing on the main screen for the course.

Example

The course shows a 40%

The debugger for cloud shows a 60% for cmi.core.score.raw

Thanks, Nicholas. I reached out to Articulate with the same message at the same time that I posted on Trivantis, simply because I couldn't figure out who should get the trouble ticket ... you responded first. Articulate sent e-mail yesterday suggesting that I send them the the files as well. I'm putting that together for submission now.

I'll let you know what they offer.

Question: does tech support at Trivantis collaborate with Artculate on issues like this?

Thanks, Dave

Hi Nicholas, I tried to emulate your 3rd test, but each time I ran it in SCORM Cloud, it reported the score of the latest attempt.

undefined

Also, when you wrote,"Note SCORM cloud does work with the mastery score and CourseMill does" it almost seems like there's a missing "not" somewhere. Can you verify?

Also, have you looked at your CourseMill transcript on this? It seems to me that each attemp (interaction) should be divisible by 20, since all the questions are worth 20. I've attached a screen shot showing mine.

What do you suggest for next steps?

Dave

undefined

If you would like help on a conference call with Articulate, I can be available for one. Just submit a Trivantis ticket and say you where working with Nick

Hello Nick, I just heard back from Articulate Support. I'm copying their e-mail and my reply here. As I'm "the guy in the middle" I'm asking for advice on what to do next. Message follows:

Dave

Thanks Angelo. I will pass your question on to Trivantis.

However, keep in mind that the Articulate 09 version works as expected in the CourseMill LMS. The '13 version does not. That tells me that there's a difference in the published SCORM 1.2 output between the two versions. Because of that, this can't be set aside as solely a CourseMill issue, don't you think?

I will let you know how Trivantis responds. The tech that I'm working with there is willing to hold a conference call to help figure this out, if need be.

Best Regards,

Dave Neuweiler

561-459-5445

-----Original Message-----

From: Angelo Cruz, Articulate Support [mailto:support@articulate.com]

Sent: Tuesday, September 13, 2016 14:40

To: dneuweiler@mms-lite.com

Subject: RE: Case #00886394: Description in this string: https://community.articulate.com/... [ ref:_00D30Txo._5003310AEKr:ref ]

Hi Dave,

Thanks for allowing us time to review the Debug logs. We're seeing quite a few instances of "The data model element in question was not implemented." And you have also mentioned seeing error 401 Undefined Data Model Element (401).

Our recommendation would be to get back with CourseMill LMS support and ask why they are returning that error so often. Specifically, what data items do they not like us sending? Maybe that the published version is for a SCORM that they don't support?

If CourseMill can start accepting the data we send or recommend to use a different SCORM version, this will be easier to diagnose.

You may also want to look into this article for help in working with your LMS:

https://www.articulate.com/support/storyline-2/troubleshooting-lms-issues

That should cover the basics including checking with the LMS to see that they are using a supported e-learning specification.

Let me know if you need anything else.

Let me know if you need anything else!

Angelo Cruz

Articulate Support - http://www.articulate.com/support/

undefined

We both agree that we don't know what that is checking for, and to be honest it's not a very detailed error. In most Course authoring tools, this type of error will point to the a item, like the timing information. If we knew what item the course doesn't like i would be able to tell you and Articulate what formats we except for that item and where to find that information in the SCORM ALD documentation.

I had another SCORM expert take a look at this just to be certain. It looks like the course is recording proper scores and then when you choose to retake the course a second time, the score is set to 100% and then to 0%. Not sure why that would be the case but that is what we are seeing.

With CourseMill we have the Manage Property "Retain highest score" set to yes by default. This means we will keep the 100% and ignore the 0%. We also support the use of a Mastery Score (which your course uses), so when we get a 100%, your course will get a Passed status.

Please let me know if this helps.

Thanks,

Nicholas Haven

undefined

The issue with the AICC version is that you have an apostrophe int he name of the SCO. Just remove the apostrophe from the name of the sco and the name of the course and republish to AICC.

Thanks for the update; I've passed it on to Articulate Support.

Did the other SCORM expert have anything to say about the numerous errors in the Debug Logs?:

"We're seeing quite a few instances of "The data model element in question was not implemented." And you have also mentioned seeing error 401 Undefined Data Model Element (401)."

Thanks, and I'll let you how Articulate responds.

Dave

Good morning. I tried a different way, by publishing to AICC instead of SCORM 1.2. Unfortunately, I cannot open the course to test it.

I've attached screenshots from CourseMill to show what I'm seeing. (On the "access denied" shot, you'll see the course ID is dn-02, while the others are dn-01. I loaded a separate iteration to manually correct the address in the Course Item Details, so this is not a typo.)

I've also attached the AICC Package.

Nick, should I be opening a regular support ticket on these issues? Let me know!

Thanks, Dave

The cause of the errors could be that Coursemill doesn't support LMSSetValue calls for the cmi.core.score.min and cmi.core.score.max values that are being set within the content. I am not a Coursemill expert, by any means, so that may not be the case. However, I took a look at the content with Nicholas and it seemed that the content would set these values each time a score was set, which caused the 401 errors in the debug of the course.

Good morning Nicholas. Here's some updated information (I'm just copying an e-mail from Articulate with my reply):

Thanks, Angelo.

undefined

In the meantime, I’ll forward your question about the “Not implemented error (401)” errors to Trivantis.

undefined

Although I have an apparent solution to the issue in AICC, I think it’s in the best interests of both Articulate and Trivantis to sort out the SCORM 1.2 issue. So if either Articulate and/or Trivantis wants to keep working on this, I’m happy to work with both.

undefined

561-459-5445

undefined

-----Original Message-----

From: Angelo Cruz, Articulate Support [mailto:support@articulate.com]

Sent: Thursday, September 15, 2016 17:58

To: dneuweiler@mms-lite.com

Subject: RE: Case #00886394: Description in this string: https://community.articulate.com/... [ ref:_00D30Txo._5003310AEKr:ref ]

undefined

1) Remove special character in quiz title:

undefined

https://www.articulate.com/support/quizmaker-09/the-quiz-title-in-quizmaker-09-should-not-include-special-characters

undefined

2) On the 09 file that worked, did you try upgrading the file to '13 format to see if it will work?

undefined

3) On the Coursemill FAQ online, AICC is also supported in addition to SCORM 1.2. Have you tried publishing to AICC to see if the issue is specific to SCORM 1.2 of Studio '13?

undefined

4) We see a lot of 401 errors according to SCORM run-time reference (http://scorm.com/scorm-explained/technical-scorm/run-time/run-time-reference/) this error is defined as:

undefined

Not implemented error (401) The data model element indicated in a call to LMSGetValue or LMSSetValue is valid but was not implemented by this LMS. SCORM 1.2 defines a set of data model elements as being optional for an LMS to implement.

undefined

Hello Ed, and thanks for joining the conversation. I passed your comment along to Articulate support, and here's their reply:

Hi Dave,

It would seem that it may be a limitation of CourseMill that they might not support LMSSetValue calls for the cmi.core.score.min and cmi.core.score.max values.

Are they able to specify or advise which values does CourseMill prefer? This is so that we can check if there's anything we can modify or settings we can possibly change in your project file to make it work with you LMS?

Please send us an email once you have an update.

Thanks,

Angelo Cruz

Articulate Support - http://www.articulate.com/support/

Along with this, I have a question. You're obviously a SCORM expert, but you mention that you're not a CourseMill expert. Is it possible to find and enlist the support of a CourseMill expert? This seems a logical step, don't you think?

Dave